An AR interface to assist human agents during critical missions

In recent years, computer scientists and roboticists have developed a variety of technological tools to aid human agents during critical missions, such as military operations or search and rescue efforts. Unmanned aerial vehicles (UAVs) have proved to be particularly valuable in these cases, as they can often enter remote or dangerous areas that are inaccessible to humans.

Researchers at Polytechnique Montréal recently developed a new system to control UAVs deployed during critical missions. This system, presented in a paper pre-published on ArXiv, is based on an augmented reality (AR) interface that allows users to control the UAVs via a head-mounted display (HMD).

"Our study was born from a partnership between the industrial company Humanitas Solutions Inc. and the research laboratory of computer graphics and virtual reality (LIRV) of Polytechnique Montréal directed by professor Benoit Ozell," Dany Naser Addin, co-author of the recent paper, told TechXplore via email. "I was a student in a research master and this paper is a result from my studies in the past 2 years."

The overall objective of the research conducted by Naser Addin and his advisor Benoit Ozell was to explore the potential of new technologies, particularly AR, for assisting humans in critical situations. The researchers were offered the opportunity to closely collaborate with people who periodically interview firefighters in Montréal (SIM – Service incendie de Montréal), in order to understand how new technology could assist them in their work.

"The goal of our study was to support the work of these firefighters in Montréal by managing a swarm of multiple drones using a single AR headset during a fire-related emergency," Naser Addin said. "To do this, we designed an AR interface, using the Magic Leap 1 headset, which can be used to manage a swarm of UAVs in a stressful situation. Our goal was to evaluate if AR could be an important tool for the future of critical situations."

The user interface (UI) designed by Naser Addin and Ozell presents contextual information related to a fire and its location right in front of a user's eyes. This information is displayed in the form of a 3D environment, which is projected on top of what a user is actually seeing at any given moment (i.e., with his/her real vision).

The system devised by the researchers ultimately allows users to control a swarm of drones in real-time via the Magic Leap 1 headset. To monitor and control the drones during critical missions, users simply need to interact with the 3D environment presented to them via the headset.

"The technology we developed can bring a huge flow of information that can overload the user and must thus be filtered in an optimal way, in order to improve the situational awareness of the user and help him/her to understand the current situation effectively," Naser Addin said.

In contrast with previously proposed solutions for controlling UAVs during critical missions, the system proposed by Naser Addin and Ozell is hands-free. This means that it allows users to focus on their vision, rather than having to simultaneously use their hands and visually monitor the situation.

The researchers evaluated their system in a series of experiments, where they asked people to tackle a complex and critical mission either using the headset they provided or a desktop computer. Their findings highlighted the benefits of AR technology in critical situations and confirmed the potential of the UI they developed.

"Emergent technology brings new research, tests, and experiments to improve some use cases, in order to reduce the complexity of tasks for humans," Naser Addin said. "AR is common today with mobile for multiple kinds of marketing or entertainment purpose. Using it with headsets in the field or for practical applications could be a huge improvement. For example, the U.S.A. want to equip their military resources with this device."

In the future, the AR-based system developed by Naser Addin and Ozell could assist human agents during a wide range of critical missions, allowing them to control UAV swarms with their vision, without having to type on a computer or use conventional controllers. If combined with infrared technology, it could also be used by firefighters or armed forces to monitor their surroundings and control UAVs when they are unable to view their environment (e.g., when they are surrounded by smoke originating from a fire or explosion).

"Unfortunately, due to the current pandemic situation, the test and deployment of our application with firefighters were postponed, so we were forced to adapt our experiments to the current situation," Naser Addin said. "Once the pandemic is over, we intend to conduct some tests with firefighters. Of course, we will also continue to research and develop similar applications of AR technology in various fields of applications such as healthcare, surgery, airplane virtual cockpits, and other collaboration environments."