Researchers use ChatGPT for choreographies with flying robots

Prof. Angela Schoellig from the Technical University of Munich (TUM) uses ChatGPT to develop choreographies for swarms of drones to perform along to music. An additional safety filter prevents mid-air collisions. The researcher's results demonstrate the first time that large language models (LLMs) such as ChatGPT can be used in robotics.

The study is published in the proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA).

The web interface is easy to use: Doctoral student Martin Schuck selects a music track and enters a text requesting a suggested choreography. Additional instructions can be given to the drone swarm via another prompt in the ChatGPT tool before an algorithm checks whether the suggested flight paths are feasible.

The screen in the Learning Systems and Robotics Lab now shows a simulated airfield with six drones flying in circles to the music. If the scientist from the Chair of Safety, Performance and Reliability for Learning Systems at TUM likes this choreography, he logs it in. Soon afterwards, six palm-sized drones take off from the floor of the robotics lab.

Choreography with three flying robots is completed within five minutes

In Prof. Angela Schoellig's laboratory, the research team has installed six ceiling-mounted cameras in a room measuring about 40 square meters and 3 meters high. Crosses are marked on the floor with insulating tape. These indicate the starting positions of the drones. Once the computer has verified a possible choreography, they can take off.

The cameras detect the position of the quadrocopters, which are equipped with four propellers and motors, 200 times a second. The system compares them with the desired position. The "airshows" in the Learning Systems and Robotics Lab, which the research team realizes with up to nine drones, are 100% safe today. Without the special safety filter, only 1 in 4 demonstrations is accident-free.

SwarmGPT: Additional algorithm ensures safety of flying robots

Prof. Angela Schoellig combined ChatGPT with the safety filter for the "Dance of the Flying Robots." "The ChatGPT AI tool was primarily created to generate texts, but it can also suggest choreographies," says the professor. "However, it initially knows nothing about the properties of drones and physical limits for the flight paths. So it is clear that ChatGPT makes mistakes."

The additional safety algorithm closes this gap by mapping out flight paths for the proposed choreography so precisely that mid-air collisions are completely avoided. Drones can even approach each other diagonally. Prof. Schoellig calls the overall concept of ChatGPT and security filter designed to use several flying robots "SwarmGPT." The tool generates the processes in the air and at the same time serves as an interface between the robot and the human, who does not require any expert knowledge.

A quantum leap through ChatGPT

When Angela Schoellig began her drone research almost 15 years ago, choreographies were hand-crafted. It took more than three years to develop the first six choreographies for six drones and get them up and running. "ChatGPT has brought about a quantum leap," says Prof Schoellig.

Over the past three months, the researchers have experimented with more than 30 choreographies for up to nine drones. Today, it takes the researchers only around five minutes to develop a safe choreography for 30-second music clips with three drones. The more drones are added, the more time ChatGPT needs for its calculations and the longer it takes to propose a choreography. But Schoellig is certain: "The concept is scalable."

Robotics: ChatGPT becomes an interface for non-experts

Can other robots be used with a similar interface via ChatGPT? For robots that use voice control to pick up objects, lay cables or open doors, the success rate of those actions is currently only 63%, 56%, and 80%. So far, applications in other robotics scenarios have been somewhat unreliable.

For Prof. Angela Schoellig, this is an incentive: "I assume that our approach will keep getting better in other scenarios, too." It may soon be possible to reprogram suction-based and industrial robots simply by voice command without the need for expert knowledge or programming skills.