Moving humanoid robots outside research labs: The evolution of the fully immersive iCub3 avatar system

Over the past four years, a research team at the Artificial and Mechanical Intelligence (AMI) lab at the Istituto Italiano di Tecnologia (IIT-Italian Institute of Technology) in Genova (Italy) has developed advanced avatar technologies, known as the iCub3 system, in continuous testing with real-world scenarios.

The system was utilized to enable a human operator to remotely visit locations 300 km away, entertain the public at events and television appearances, and compete in the international challenge ANA Avatar XPrize.

In their research paper, published in Science Robotics, the group highlights the challenges they faced, and the solutions implemented to develop the avatar system. Their approach highlights how crucial is for research to extend beyond laboratory confines and addresses the difficulties posed by the real-world variability, in order to develop a robust humanoid robotic platform that can soon become part of the economic and productive system.

The iCub3 system has already evolved into a new robot, the ergoCub robot, designed to maximize acceptability within work environments.

The research group at the AMI lab at the IIT is coordinated by the Italian researcher Daniele Pucci and comprises about 50 researchers. The iCub3 avatar system is mainly composed of the robot iCub3—which is one of the last versions of the iCub humanoid robot born in IIT 20 years ago—and the wearable technologies named iFeel that track human body motions.

The avatar system has been designed to facilitate the embodiment of humanoid robots by human operators encompassing aspects such as locomotion, manipulation, voice, and facial expressions with comprehensive sensory feedback including visual, auditory, haptic, weight, and touch modalities. It has been developed by IIT in collaboration with the Italian National Institute for Insurance against Accidents at Work (INAIL).

The IIT research group tested and tailored the iCub3 avatar system in three different real scenarios.

In November 2021, during the first test a human operator located in Genoa controlled the avatar at the Biennale di Venezia in Venice (approximately 300 kilometers away) having the possibility to visit remotely the Italian art exhibition. The challenges were about assuring stable remote communications between the two sites, within a limited testing timeframe.

Furthermore, the robot's movement and interaction with the surrounding environment had to be cautious and safe, given the delicate nature of the art exhibitions.

Pucci's group utilized the IIT wearable iFeel suit to track the operator's body motions, and the avatar system transferred these movements onto the iCub3 robot in Venice, allowing it to replicate the user's actions in Genova.

The operator exercised precise control over the robot's body parts, including the fingers and eyelids. Additionally, custom haptic devices were employed to allow the operator to feel a hug from the remote location. The transmission was streamed over a standard optical fiber internet connection, resulting in around a hundred milliseconds of delay.

In June 2022, the second test took place at the Italian digital innovation festival, We Make Future Show, presenting additional challenges.

The operator in Genova controlled a robot positioned at the festival venue in Rimini (approximately 300 km away). The robot's task involved receiving and transporting a payload from a person while navigating the theater stage before an audience of around 2,000 spectators.

The performance was conducted live, amidst considerable electromagnetic interference. The IIT research group introduced specialized haptic devices to enable the operator to perceive the weight carried by the robot, and enhanced the robot's expressive abilities, as it had to entertain the audience.

In November 2022, the entire architecture was presented at the All Nippon Airways (ANA) Avatar XPrize competition in Los Angeles. The contest arena resembled a panorama taken from Mars, or another remote planet, suggesting space exploration as a future potential application of avatars.

Each avatar was operated by individuals who were not part of the research team and required training just before the race; additionally, they had only 25 minutes to complete 10 different tasks.

The iCub3 robot had to perform heavy-duty tasks within a limited time, a situation that pushed the robot to its limits. The IIT team equipped the robot's hands with sensorized skin to allow it to perceive the texture of manipulated objects.

The robot control was designed to be as intuitive as possible, allowing the operator to have direct control over all robot parts and capabilities. For example, the operator could control the robot's locomotion by walking in place, creating a heightened sense of embodiment.

The experience acquired by Daniele Pucci's team was fundamental for turning the iCub3 avatar system into a new robot, the ergoCub robot, designed to maximize acceptability within the work environment. The new humanoid ergoCub, currently being developed at IIT, is meant to minimize risk and fatigue in collaborative tasks for workers for industry and health care.