This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:


trusted source


A deep learning and model predictive control framework to control quadrotors and agile robots

A deep learning and MPC framework to control quadrotors and agile robots
Real-time Neural MPC can, for example, be used to efficiently model highly complex aerodynamic ground effects occurring in close proximity flight to obstacles (table), using only onboard computation. Credit: Salzmann et al

In recent years, computer scientists have developed increasingly advanced algorithms for controlling the movements of robotic agents. These include model predictive control (MPC) techniques, which use a model of the agent's dynamics to optimize its future behavior toward a given goal while simultaneously satisfying a number of constraints (e.g., cannot crash into obstacles).

Researchers at Technical University of Munich and University of Zürich recently created Real-time Neural MPC, a framework that integrates complex model architectures based on (ANNs) in an MPC framework for agile robots (i.e., quadrotors—drones with four rotors). This concept, presented in IEEE Robotics and Automation Letters, builds on a previous approach developed by the University of Zurich's Robotics and Perception Group.

"We came across the excellent work by the Robotics and Perception Group led by Davide Scaramuzza, and were immediately captivated to build upon their core idea of having data-driven (learned) components boosting 'traditional' ," Tim Salzmann and Markus Ryll, researchers at the Autonomous Aerial Systems Group of Technical University of Munich, told Tech Xplore.

"After we worked out a proof-of-concept to generalize their approach, which uses Gaussian Processes (GPs), to generic Neural Networks (Deep Learning Models), we pitched our idea to the Robotics and Perception Group at the University of Zurich. From there on, technical work and experiments were jointly advanced between the two labs, sparking a new collaboration."

The new framework proposed by Salzmann, Ryll and their colleagues combines and online optimization of MPC. Expressive models are computationally heavy. However, by approximating them online in real-time the framework can leverage dedicated hardware (GPUs) to efficiently process these models, allowing their system to predict optimal actions for robots in real-time.

"The Real-time Neural MPC framework allows for the combination of two fields, optimal control, and deep learning while allowing for both parts to leverage their respective highly optimized frameworks and computational devices," Salzmann and Ryll explained. "As such, we can perform deep learning computations in PyTorch/Tensorflow on a GPU while the control optimization is performed in compiled C code on a CPU. This allows using the power of deep learning in previously unachievable applications—for example in onboard optimal control of a Quadrotor."

The researchers evaluate their framework in a series of experiments, both in a simulated and real-world environment. In these tests, they specifically use it to control the actions of a highly agile quadrotor in real-time.

Their results are very promising, as they can leverage the predictive power of neural network architectures with a parametric capacity more than 4,000 times larger than those previously used to control the movements of agile robots in real-time. They also find that compared to conventional MPC methods without a deep learning component, their framework can reduce positional tracking errors by up to 82%.

"In robotics, we seek expressive models of the dynamics of the controlled systems and their interaction with the environment (e.g., aerodynamic effects, tire friction, etc.)," Salzmann and Ryll said. "While these are often hard to formulate analytically, learning-based approaches, especially neural networks, can capture dynamics and interaction effects. However, the model accuracy scales with the size of the Neural Network. Real-time Neural MPC enables the use of deep learning models many magnitudes more powerful and performant compared to what was previously feasible in Model Predictive Control."

As demonstrated by the recently released Nvidia Jetson platform, GPU chips are gradually finding their way into embedded systems. The framework created by this team of researchers could soon allow developers to leverage the high predictive power of advanced data-driven AI techniques to better model the dynamics of robots integrating GPU chips and their interactions with the environment, reducing the risk of accidents and improving their navigation capabilities.

"There are many open avenues for future research," Salzmann and Ryll added. "One problem of deep learning approaches is that their output can be erratic for situations that are not part of the training data (Out Of Distribution OOD). Detecting these situations and providing a fallback for the control to stabilize the system would lead to robustness in OOD situations."

More information: Tim Salzmann et al, Real-time Neural MPC: Deep Learning Model Predictive Control for Quadrotors and Agile Robotic Platforms, IEEE Robotics and Automation Letters (2023). DOI: 10.1109/LRA.2023.3246839

Guillem Torrente et al, Data-Driven MPC for Quadrotors, IEEE Robotics and Automation Letters (2021). DOI: 10.1109/LRA.2021.3061307

© 2023 Science X Network

Citation: A deep learning and model predictive control framework to control quadrotors and agile robots (2023, March 13) retrieved 23 March 2023 from
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Explore further

A method to enable robotic paper folding based on deep learning and physics simulations


Feedback to editors