New system enables intuitive teleoperation of a robotic manipulator in real-time

Imitation learning is a promising method to teach robots how to reliably complete everyday tasks, such as washing dishes or cooking. Despite their potential, imitation learning frameworks rely on detailed human demonstrations, which should include data that can help to reproduce specific movements using robotic systems.

One way to collect these demonstrations is to use teleoperated systems, which allow humans to control the movements of robotic manipulators to complete specific tasks. Most existing teleoperation systems, however, struggle to effectively process reproduce the complex and coordinated movements performed by humans.

Researchers at University of California, San Diego recently developed Bunny-VisionPro, a system that enables the teleoperation of a robotic system to complete bimanual dexterous manipulation tasks. This system, presented in a paper posted to the arXiv preprint server, could facilitate the collection of human demonstrations for imitation learning.

"Our recent paper was inspired by the need to advance bimanual dexterous teleoperation for robotics," Xiaolong Wang, co-author of the paper, told Tech Xplore.

"We recognized that existing vision-based teleoperation systems rarely focus on dual-hand control, which is crucial for complex tasks requiring hand coordination."

The primary objective of the recent work by Wang and his colleagues was to develop a general teleoperation system that could generalize well across different types of robots and tasks, greatly simplifying the collection of demonstrations to train robotics control algorithms.

The system they developed, called Bunny-VisionPro, could make teleoperation and the collection of demonstration data as intuitive and immersive as playing a game in virtual reality (VR).

"Bunny-VisionPro is an innovative bimanual dexterous teleoperation system that allows human operators to control dual robot arms and multi-fingered hands in real time," Wang explained. "It enables high-quality demonstration collection to boost imitation learning."

The device teleoperated via the team's system is portable and lightweight. While users are teleoperating this device, they also receive visual and haptic feedback, which makes their experiences more immersive.

In initial tests, the Bunny-VisionPro system was found to enable the safe control of the robotic manipulator in real-time and with low latency, while also preventing collisions with nearby objects. Notably, the system is also easy to install and deploy in laboratory settings, making it a promising solution for collecting demonstrations in robotics labs.

Bunny-VisionPro has three key components: an arm motion control module, a hand and motion retargeting module and a haptic feedback module.

"The arm motion control module retargets the human wrist pose to the robot's end-effector pose, handling singularity and collision issues," Wang said. "The hand and motion retargeting module maps human hand poses to robot hand poses, including support for loop-joint structures. Finally, the haptic feedback module transfers robot tactile sensing to human haptic feedback through algorithmic and hardware design."

The primary advantage of the new teleoperation system developed by Wang and is colleagues is that it enables the safe control of a bimanual robotic system in real-time. In contrast with several other solutions introduced in the past, it also integrates haptics and visual feedback, which makes the collection of demonstrators more immersive for human users, while also improving the system's teleoperation success rates.

"The real-time bimanual dexterous teleoperation system we developed effectively balances safety and performance," Wang said. "This system enables control of robot arms and multi-fingered hands with minimal delay, while incorporating collision avoidance and singularity handling. This is significant for working reliably in real-world robotic applications."

This recent work by Wang and his colleagues could greatly simplify the use of teleoperation for collecting demonstrations for imitation learning frameworks. In the future, Bunny-VisionPro could be deployed at other robotics labs worldwide and could inspire the development of similar immersive robotic teleoperation systems.

"In our next studies, we plan to further enhance manipulation capabilities by leveraging the robot's tactile information for better precision and adaptability," Wang added. "The code for our system is open-source and can be accessed on GitHub."

© 2024 Science X Network