Examples of tasks performed during the CST: (a) physical exercise (b) memory training (c) positive feedback. Credit: Castellano et al.

Robots are gradually being introduced in a wide range of real-world settings, including malls, manufacturing facilities, and healthcare facilities. A way in which robots could be particularly useful is in assisting seniors in both their homes and elderly care facilities.

Researchers at University of Bari and University of Parma have recently published a new study exploring the emotional reactions of a small group of seniors after they interacted with Pepper, a humanoid robotic system. Their paper, available in Springer Link's Human-Friendly Robotics 2021, suggests that seniors can display both negative and while interacting with robots.

"We previously conducted other studies with the NAO in the context of Cognitive Stimulation Therapy (CTS) and we measured not only the performance of seniors in terms of task completion, but also in terms of engagement and affective response," Berardina De Carolis, one of the researchers who carried out the study, told TechXplore. "Our results were promising. When Pepper arrived in our department, we noticed that some of its features could overcome the limitations of the NAO robot (i.e., the display, the height, the possibility to move in a room, etc.) and produce better results."

In their previous studies, the researchers had investigated the impact of using NAO, a programmable humanoid robotic platform, to perform CST on senior adults. CTS is a form of therapy aimed at enhancing the mental skills of older adults or younger patients with memory and cognitive impairments. In one of their studies, the team specifically looked at how acceptant the patients were of the robots and on how well they performed on tasks.

"For instance, we used NAO H25 in a healthcare center for cognitive disorders and dementia, as model in demonstrating not only physical exercises to a group of seniors, but also in groups therapy sessions to assist the therapist with recovering and/or maintaining such as memory, orientation and providing to the participants instructions, suggestions, and consequences," Olimpia Pino, another researcher involved in the studies, told Tech Xplore.

After their lab received the Pepper robot, De Carolis, Pino and their colleagues decided to conduct a new investigation at the Alzheimer's center in Bari, adapting the tasks they used in their previous work to leverage the new robot's more advanced communication abilities. Due to the COVID-19 pandemic, however, they were forced to reduce their sample of participants and conclude their experiment earlier. Nonetheless, their study provides a general indication of how seniors might react to interactions with the Pepper robot and to CST sessions assisted by the robot.

A CST session with Pepper. Credit: Castellano et al.

"The progressive use of humanoid robots in assistive or therapeutic functions raised several issues concerning participants' engagement and acceptance," Pino explained. "In our previous research, when we chose experimental tasks such as prose recall, calculations, matching song-singer, we noticed that robots made tasks more engaging, and, in some circumstances, patients reacted enthusiastically to their interactions with the robots. Moreover, most participants indicated they would like to have the robot at home."

In their new study, De Carolis, Pino and their colleagues analyzed the emotions of 8 seniors while they completed memory tasks with support from the Pepper robot. To do this, they compared the predictions made by an automatic Facial Expression Recognition (FER) system specifically trained to detect emotions in the faces of older adults with observations made by three human raters. The Pepper robot proved to be particularly well-suited for their activities, as it features tactile sensors on its head and hands, as well as a tablet on which it can play videos or images, and buttons that users can press to interact with the robot.

"Despite many difficulties in programming the robot and the consequent still too limited ability to appear natural, it has been observed that elders became more engaged with Pepper along sessions showing more positive facial expressions," Pino explained. "It is desirable that robots applied to real world applications perform their activities in reactive but flexible manner. Thus, robots capable to adapt to human interaction are particularly suitable."

De Carolis, Pino and her colleagues found that participants exhibited both negative and positive emotions while interacting with the robot on memory tasks. Interestingly, human annotators observed more positive emotions than the automatic emotion recognition system did. Nonetheless, most of the emotional reactions of seniors appeared to be positive.

"Humanoid robots seem promising since they can support more engaging interactions with users, and results obtained so far are encouraging," Pino said. "The integration of robotics into both formal and informal care opens new possibilities for improving the lives of patients and alleviating the burden on caregivers and healthcare services. However, to make the FER model more robust in real-time and in the wild we will need to explore the use of deep learning."

The recent findings gathered by this team of researchers further confirm the potential of introducing robots in elderly care facilities and other therapeutic settings for older adults. In their next studies, De Carolis, Pino and her colleagues plan to gather a larger dataset with senior facial expressions, to develop and train a deep learning method that can predict their participant's emotions with greater accuracy.

"In our future research, we also want to personalize the interventions and tasks to meet the needs of individual patience and, since we developed a module for analyzing the seniors' behavior in terms of speech, gaze and emotions, we want to endow the robot with the possibility to react and adapt its answers to the recognized senior's feedback," De Carolis added.

More information: Giovanna Castellano et al, Detecting Emotions During Cognitive Stimulation Training with the Pepper Robot, Human-Friendly Robotics 2021 (2022). DOI: 10.1007/978-3-030-96359-0_5