A bite acquisition framework for robot-assisted feeding systems

According to a survey released by the U.S. Census Bureau, around 12.3 million Americans require assistance with activities of daily living (ADLs) or instrumental activities of daily living (IADLs), one of which is feeding. Robots could be of great help to people affected by severe disabilities, allowing them to eat meals and complete other daily tasks without reliance on constant assistance from other human beings.

With this in mind, a team of researchers led by Prof. Siddhartha Srinivasa at the University of Washington's Personal Robotics Lab has been trying to develop a robot-assisted feeding system that can automatically pick up food from plates and feed it to human users. In a recent paper pre-published on arXiv, the researchers introduced a bite acquisition framework designed to calculate and attain reasonable "bites" of food from a plate or bowl.

"As a lab, we are passionate about developing robots that can assist people in their day to day lives," Tapomayukh Bhattacharjee, one of the researchers who carried out the study, told TechXplore. "Through this project, we want to develop robots that can feed people autonomously. To feed people a wide variety of food items, a robot needs to have the capability to acquire previously unseen food items. In this paper, we focus on the problem of bite acquisition of previously unseen food items."

A key challenge when developing robot-assisted feeding systems is ensuring that these robots can effectively pick up all the types of food they encounter. This can be difficult to achieve, as different food items have a variety of physical properties and thus require different acquisition strategies. Ideally, a robot-assisted feeding system should be able to pick up any food item on a plate, even if it has never encountered it before.

To better understand what acquisition strategies work best for particular types of food, Bhattacharjee and his colleagues collected data from 2450 robot bite acquisition trials using 16 food items with varying properties. When they analyzed this data, they realized that items with similar physical properties exhibit similar acquisition success rates, which makes it easier to generalize an acquisition strategy to previously unseen items. Their analyses also offered insight into how other factors (e.g. the environment surrounding the food, fork pitch, fork angle, etc.) can affect a robot's success in picking up a bite of food from a plate.

Based on these observations, the researchers developed a bite acquisition framework that uses two distinct neural networks in a hierarchical structure. The first network, called RetinaNet, analyzes full-plate images containing different types of food and then outputs bounding boxes around individual items. The second network, SPANet, uses these bounding boxes to calculate the success probability for different bite acquisition actions and the skewering axis for each food item.

"To output the success probability, SPANet also uses features related to the surrounding environment of a food item, as we found that the surrounding environment affects a robot's choice of actions as well as the success rate," Bhattacharjee explained. "To encode the environment features, we developed an environment classifier that identifies items as being in one of three environments: isolated, near a plate edge or another food item, or on top of other food items."

The researchers applied their framework to a JACO robotic arm and evaluated its performance in a series of experiments on uncluttered and cluttered plates containing various food items. These tests yielded very promising results, with their method successfully generalizing skewering strategies across previously unseen food items.

"Our network, SPANet, could successfully generalize the actions to previously unseen food items with similar bite acquisition action distributions," Bhattacharjee said. "Practically, this means that given a new food item that the robot hasn't seen before, our network should be able to infer how to acquire it successfully from a plate/bowl if the new food item can be acquired by similar actions."

In the future, the bite acquisition framework developed by Bhattacharjee and his colleagues could aid the development of more efficient robot-assisted feeding systems. Meanwhile, the researchers plan to extend their approach's bite acquisition actions, as this would allow robots powered by their framework to pick up an even wider variety of food, including items such as rice and mashed potato.

"We are also interested in exploring ways to acquire previously unseen food items which require very different actions for picking them up compared to what the robot has seen before," Bhattacharjee said.

© 2019 Science X Network