A novel motion-capture system with robotic marker that could enhance human-robot interactions

Motion capture (mocap) systems, technologies that can detect and record the movements of humans, animals and objects, are widely used in various settings. For instance, they have been used to shoot movies, to create animations with realistic lip and body movements, in interactive videogame consoles, or even to control robots.

Researchers at Skolkovo Institute of Science and Technology (Skoltech) in Russia recently developed a new motion-capture system that could enable safer and more effective human-robot interactions. This system, presented in a paper pre-published on arXiv, is based on a wearable robotic system and camera to track the movements of a user's arm.

"This work was the result of many years of research in the Human-Robot interaction domain at the Intelligent Space Robotics (ISR) Lab," Ali Alabbas, the key researcher for the project, told Tech Xplore. "The Human-Robot interaction deals with collaborative tasks that both humans and robots take part in, so the safety of the human operator is a top priority in such applications. The work was built on the previous experiences and technologies developed in our Lab."

The mocap system introduced by this team of researchers draws inspiration from two previous technologies they developed. The first is a system called CobotGear, which uses a 6-DoF wearable camera to track the human arm motion. The second, CoHaptics, leverages haptic (touch-related) feedback, to give users information about their distance to a robot and alert them when a robot is critically close to them.

"One objective of our work is to develop a low-cost and easy-to-install mocap system that detects and localizes the user's hand in the work environment, employing the algorithm underpinning the functioning of CobotGear to avoid collisions between the robot and the user as they are engaging in collaborative tasks," Alabbas explained. "The proposed mocap system is vision based. It uses cameras and computer vision to locate a unique visual robotic marker attached to the user's hand (the target for our mocap system)."

A common limitation of mocap systems is that they can be adversely affected by occlusions. In other words, if a visual marker that the system is following is partially or totally hidden behind an object, the system is no longer able to track it. Alabbas and his colleagues set out to address this limitation, by reducing the risk of occlusion.

"Our idea was to develop a mechanism that could move the marker, ensuring that it always remains visible to the mocap system," Alabbas said. "Our solution was to build a small wearable robot that would hold the marker and have the ability to rotate."

The system created by this team of researchers is based on a robot that users should wear on their forearm. This robot moves a marker in different ways, making sure that it always remains in the field of view of an integrated camera.

"Typically, retroreflective markers of mocap have a spherical form because the size will be the same if captured by cameras from different perspectives," Professor Dzmitry Tsetserukou said. "On the other hand, you need to place many of the markers on the human body and to install expensive IR cameras to cancel all visible objects except spheres, thus mocap systems can cost hundreds of thousands of dollars. We have proposed ArUcoGlide, an affordable and robust solution to track human body motion with low-cost single camera and to even talk with the user motion with tactile cues."

The ArUcoGlide robot is extremely light, weighing no more than a simple wristwatch. This means that it can be easily worn by users, without causing them discomfort, The robotic system holds a unique marker that is located by a camera, providing valuable feedback aimed at preventing collisions during human-robot interactions.

"The system consists of three parts: a wearable robotic marker, a tracking system and a collision avoidance controller," Alabbas said. "The wearable robot, which we dubbed ArUcoGlide holds a unique visual marker and adjusts this marker's orientation to ensure its visibility to the camera and to address occlusion by objects within the environment. The robot can also provide haptic feedback in the form of vibration in the event of a hazardous situation. We suggested that maintaining a fixed orientation between the camera and the marker will reduce the moments when the marker is not shown because of the movement of the user."

The second component of the team's mocap solution is a tracking system, consisting of a camera and a base computer. The camera captures a live stream of the environment in which humans are collaborating with robots. This stream is then processed by the base computer to extract the coordinates of the marker within the environment.

"The third component, the collision avoidance controller, was first introduced as part of our previously developed CobotGear system," Alabbas said. "This controller governs the movement of the robot to avoid potential collisions with the user's hand in the workspace."

Essentially, the team's system works as follows. A camera records the environment in which humans and robots are collaborating. The AucoGlide robot adjusts the orientation of the marker used to track motions, ensuring that it always remains within the camera's field of view. Finally, the collision controller moves the robot to prevent it from contacting with users.

"In a nutshell, marker glides in space with the 2-DoF robot to get continuous capturing of the body motion," Tsetserukou explained. "If the marker is static, i.e., it does not adjust the position in space, the tracking system will lose it at a very steep or shallow angle with respect to the camera. It is not applicable when the safety of humans is the top priority."

The researchers evaluated ArUcoGlide in a series of experiments, testing the accuracy of its tracking system, how the system affects the behavior of humans and robots just before an imminent collision, its performance during a real-world collaborative task. They found that it significantly improved the safety of robot-human interactions, maintaining increasing the distance between a user's hand and the robot to an average of 5 cm.

The system could prove effective for enhancing the human-robot collaboration in a variety of real-world settings. For instance, it could assist medical personnel who is pipetting blood samples.

"When the human arm comes close to the robot, our system can detect it and activate the collision avoidance algorithm, so that the robot does not injure the user," Tsetserukou said, "Experiments proved a sub centimeter accuracy of human arm tracking in a wide range of motion. Importantly, the system with moving ArUco markers and tactile feedback speeds up the shared medical test process up to 16%."

Remarkably, the team's mocap system can also be adapted to meet the needs of specific applications. For instance, if users wish to change the environment it is monitoring they simply need to move its motion tracking camera to the desired location and restart the system.

"The system is designed to be used in a shared work environment reminiscent of typical activities found in warehouses or laboratories," Alabbas said. "For instance, it can be installed in a laboratory where a robotic arm can join a chemist in preparing some solutions, speeding up the process and protecting the user from exposure to toxic substances," Alabbas said.

"The chemist's attention for sure is focused on the materials, so our system, by monitoring the work environment, can always locate the chemist's hand and try to avoid collision of robot with it. ArUcoGlide is very suitable because of its cost-effectiveness and ease of installation; the user just needs to fix the camera in a suitable place to detect the work environment, wear the ArUcoGlide, and then start working."

ArUcoGlide could soon be implemented and tested in a wider range of scenarios, to validate its potential for improving human-robot interactions. In their next works, Alabbas, Tserserukou and their colleagues would like to develop it further to enable the tracking of a user's entire body.

"We plan to develop GlideSuit, a system in which robotic markers are placed on the whole body: arms, hands, legs, and torso.," Tsetserukou said. "ArUco markers will move synchronously on the human body to maintain normal with the optical axis of the camera."

This updated version of ArUcoGlide could be used to track a user's skeleton in real-time using a single camera. In addition, it could support augmented reality (AR) applications, for instance allowing users to visualize organs or muscles inside the human body or to see how different items of clothing would look on their body.

"Imagine shooting a video of a moving person holding virtual 3D birds in real-time," Tsetserukou said. "ArUcoGlide can be used not only for humans but also for robots. Placed on the top of the drones, with only a single camera at the ceiling we can track a swarm of robots and control their formation for light shows. Another fascinating application is dance teaching and rehabilitation when we guide the human with tactile feedback to get the desirable limb trajectory in a 3D environment."

© 2023 Science X Network