Figure outlining how the human-like planner works. Credit: Hasan et al.

While research in the field of robotics has led to significant advances over the past few years, there are still substantial differences in how humans and robots handle objects. In fact, even the most sophisticated robots developed so far struggle to match the object manipulation skills of the average toddler. One particular aspect of object manipulation that most robots have not yet mastered is reaching and grasping for specific objects in a cluttered environment.

To overcome this limitation, as part of an EPSRC-funded project, researchers at the University of Leeds have recently developed a human-like robotic that combines virtual reality (VR) and machine learning (ML) techniques. This new planner, introduced in a paper pre-published on arXiv and set to be presented at the International Conference on Robotics and Automation (ICRA), could enhance the performance of a variety of robots in manipulation tasks.

"Our research goal is to develop better robotic systems; systems that can help humans in a range of tasks, ranging from exploring hazardous environments to helping a child learn to write," Prof. Anthony G. Cohn, principal investigator for the study, told TechXplore. "So we formed a multidisciplinary group of psychologists and computer scientists to explore whether we could capture the behavior of humans and reverse-engineer the rules that humans use when reaching for objects."

In contrast with other researchers who trained machine learning classifiers on images of cluttered environments, Cohn and his colleagues wanted to generate using new immersive technologies. In their study, they thus used VR equipment to collect data on humans completing manipulation tasks.

Subsequently, they used ML classifiers to analyze the VR data they collected and extract general rules that may be underlying human decision-making. The planner uses these rules to plan efficient strategies for robots completing object manipulation tasks.

"We wanted the resulting plans executed by the robot to also be 'human legible," in the sense that they would be the ones that a human would expect another human to execute, which is not the case for many current robot plans," Cohn explained.

The decisions of humans who are completing manipulation tasks appear to be mostly guided by qualitative representations (i.e., not the exact distance and direction of all objects in their surrounding environment, but rather relative distances and directions). In their study, the researchers tried to better understand these decision-making processes in order to partly reproduce them in ML classifiers.

"We explored how humans reach in scenarios where the objects they grasp have particular spatial relationships and used methods developed by AI researchers at the University of Leeds to describe qualitatively where the objects were located," Dr. Mohamed Hasan, a research fellow working on the project, explained. "This meant that the planner could classify the qualitative spatial relationships between objects and then select the actions that humans would use in that scenario."

The approach proposed by Cohn and his colleagues allows robots to identify effective actions to reach for an object in a cluttered space much faster than they would if they had to consider all possible actions. It does this by producing a high-level plan, which is represented as a sequence of key waypoints and moves. This plan is transmitted to a standard low-level planner, which uses it to plan detailed trajectories for the robot's arm.

"Our planner works in a way that resembles the process of planning a journey by choosing which towns to go through and only later deciding exactly which roads and lanes to take," Cohn said. "We found that this approach makes our human-like planner much more efficient than existing planners."

The researchers evaluated their planner in a series of experiments, testing it in VR scenarios where humans completed manipulation tasks in physics-based robot simulations and using a real robotic gripper. All three evaluations yielded very promising results, with the human-like planner outperforming a state-of-the-art, standard trajectory optimization algorithm.

The planner devised by Cohn and his colleagues was able to generate effective strategies that allowed robots to reach for objects in cluttered environments faster and more efficiently than they would when using the standard trajectory optimization techniques. In addition to introducing a promising human-like planner for robot manipulation tasks, the study shows that VR technology can be used to study human behavior and gain a better understanding of decision-making processes.

"Our VR platform, which was developed in-house at the University of Leeds, enabled us to record hundreds of reaches in a short period of time, but we could also change the layout of the environment between each trial and easily present exactly the same environment to different human participants," said Prof. Mark Mon-Williams, co-investigator and a cognitive psychologist, explained. "Our findings support our recent suggestions that VR will become one of the most ubiquitous tools in psychological research. Yet we also found that rules underpinning human action selection can be captured by ML methods."

In the future, the planner developed by Cohn and his colleagues could help to enhance the manipulation skills of both existing and new robots, allowing machines to select actions more efficiently and potentially even explain the "reasoning" behind their decisions. This recent study could also encourage other researchers to use VR data when investigating human decision-making processes, which could then be better reproduced in machines.

The ultimate goal of the research is to build intelligent robots that will understand and anticipate human intentions in human- collaboration scenarios. Ideally, such robots would also be able to respond in human-like ways, communicating with human users similarly to how another human would.

"The current work is a proof of concept, so we now plan to take the method we developed and exploit its immense potential," Mon-Williams said. "It has become clear that this combination of computer science and psychology is a very useful approach that has the potential to shed light on action selection, which is important for a number of reasons, including understanding how to treat medical conditions such as strokes, as well as help us design more efficient robots. We are currently working on our next series of experiments and projects."

More information: Human-like planning for reaching in cluttered environments, Mohamed Hasan, Matthew Warburton, Wisdom C. Agboh, Mehmet R. Dogar, Matteo Leonetti, He Wang, Faisal Mushtaq, Mark Mon-Williams, Anthony G. Cohn, arXiv: 2002.12738 [cs.RO]. arxiv.org/abs/2002.12738, set to appear ICRA-20 (the International Conference on Robotics and Automation).

artificial-intelligence.leeds. … uman-like-computing/

Studying human behaviour with virtual reality: The Unity Experiment framework. Jack Brookes, Matthew Warburton, Mshari Alghadier, Mark Mon-Williams, Faisal Mushtaq, DOI: 10.3758/s13428-019-01242-0. link.springer.com/article/10.3758%2Fs13428-019-01242-0