A fully autonomous drone system for cinematography and wildlife monitoring

Recent technological advances, such as increasingly sophisticated drones and cameras, have opened exciting new possibilities for cinematography. Most notably, film directors can now shoot scenes from a wide range of angles that were previously inaccessible and in far higher resolution.

Researchers at University of Zaragoza and Stanford University recently developed CineMPC, a new cinematographic system that relies on a fully autonomous drone that carries a cinematographic camera to film multiple targets autonomously, while following a director's instructions. The platform modulates various drone and camera parameters to satisfy these instructions. The team's innovative system, outlined in IEEE Transactions on Robotics, could bring a wave of innovation to the film industry and other sectors that can benefit from high-quality video footage.

"Existing solutions for autonomous drone cinematography revealed a common oversight, namely, none provided automatic control over camera intrinsic parameters (i.e., focal length, aperture, focus distance)," Pablo Pueyo Ramon, co-author of the paper, told Tech Xplore.

"In cinematography, controlling these parameters is essential to achieve different artistic and technical goals, such as a desired depth of field (parts of the scene shown in focus or blurred), or iconic shots like the dolly-zoom or vertigo effect. CineMPC fills this gap, determining autonomously the appropriate camera intrinsics capable of achieving a wide array of user-defined cinematographic instructions."

Pueyo and his colleagues have been developing innovative technologies for cinematography for some time now. The new system they developed, dubbed CineMPC, was first introduced in 2021.

CineMPC essentially consists of software that can be installed in any drone equipped with a controllable professional camera (e.g., a DSLR camera). As part of their recent study, the researchers broadened their software's functionalities and set out to assess its performance in a real-world setting.

"Besides including a better and more advanced control strategy, the new version of CineMPC also includes a perception module capable of identifying relevant information from the scene, like the actors and actresses, making it a 100% autonomous control solution," Pueyo explained. "Finally, we are now releasing the source code so that everybody can use it."

Pueyo and his colleagues have tested the improved version of CineMPC by applying it to a real drone for cinematography and filming various scenes with it. They found that their software achieved remarkable results, reliably estimating the relative poses of filmed targets and allowing users greater control over the footage captured by the drone, for instance adding unique effects, tracking specific people or objects, and so on.

"In our opinion, the implications of our study for cinematography alone are remarkable," Pueyo said. "We are enormously happy to offer filmmakers creative freedom, improved safety, and increased autonomy for real-time decision-making."

The updated version of CineMPC can already be readily accessed by developers cinematographers and film-makers online. Therefore, it could soon be used to enhance the use of drones for shooting movies, short-films, and documentaries.

CineMPC could also prove useful for various other applications, for instance allowing environmental scientists and researchers to closely monitor wildlife and natural environments from afar and without disrupting them, by controlling the zoom of drone-equipped cameras. In addition, it may also be a promising tool for precision agriculture, as it enables both the monitoring of large fields at a distance and the close inspection of specific plants.

"We are currently working on the development of advanced AI techniques to simplify the interaction with CineMPC, aiming to democratize the use of autonomous drones," Pueyo added. "Instead of relying on handcrafted input instructions, we plan to develop user-friendly interfaces and intuitive controls.

"For example, we have just published another paper, in which the instructions to the drone are defined by a video sequence, for example, a small cut of your favorite movie. Our solution identifies the necessary information to control the drone and the camera to reproduce it and CineMPC executes it."

© 2024 Science X Network