Gaze-controlled robotic arm for the speech and motor impaired

Eye gaze-controlled robotic arm for the speech and motor impaired
Credit: Pradipta Biswas

People with severe speech and motor impairment (SSMI), a condition caused by disorders like cerebral palsy, have difficulty physically operating devices such as a joystick, mouse or trackball, or use speech recognition systems. Eye-gaze-controlled computer interfaces can help them perform such tasks on par with their non-disabled peers.

To help people with SSMI, a research team at the Centre for Product Design and Manufacturing (CPDM), Indian Institute of Science (IISc), has designed a robotic arm that can be manipulated by eye movement using a . This is non-invasive, using a webcam and a computer, unlike other eye gaze-tracking devices that use head-mounted systems.

The team worked with young adult students who have SSMI at Vidya Sagar (formerly the Spastics Society of India) in Chennai, an institution for individuals with disabilities. "Many of these students [with ] are not able to precisely focus at a in their due to uncontrolled gaze movement. They are also not comfortable to look at all portions of the visual field equally," says project lead Pradipta Biswas, assistant professor at CPDM.

Biswas and his team used computer vision and machine learning algorithms to analyse live feeds of facial video from the users, and were able to estimate where the user was looking. They coupled this with an augmented reality application to allow the user to use a robotic arm for tasks like picking up and dropping objects, and placing them where they desire within reach of the robotic arm.

One of the main applications of this eye-gaze-controlled robotic arm is rehabilitation of young adults with SSMI through tasks such as fabric printing. They usually require assistance in doing such tasks, as they can only do a small part of it by themselves. Using the robotic arm designed by the researchers, people with SSMI can use their eye gaze to perform mechanical tasks that can help them work on handicrafts independently.

This eye-gaze-controlled interface has been deployed at Vidya Sagar and is in regular use. "This validation and evaluation [of the interface and ] with end users is a major contribution of the study," says Biswas. The students with SSMI are able to use the robotic manipulator almost as well as their non-disabled counterparts, and the results were the same during repeated trials.

This technology can also be used by younger individuals with SSMI to move toys like cars. "We are using play as a medium to teach new technologies, which they can use for the rest of their lives," adds Biswas. Further modifications to this tool could also allow young individuals with SSMI to use it for e-learning.

The authors believe that this interface is a step toward future technologies that ensure physical impairments do not stand in the way of pedagogical training and professional lives of individuals with SSMI. Biswas also says that such a system could be useful for automotive and aeronautical applications, as well as for developing collaborative robots used in smart manufacturing.

More information: Sharma, V. K., Saluja, KPS., Mollyn, V., & Biswas, P. (2020, June). Eye Gaze Controlled Robotic Arm for Persons with Severe Speech and Motor Impairment. In ACM Symposium on Eye Tracking Research and Applications (pp. 1-9).

Sharma, V. K., Murthy, LRD, Saluja, KPS., Mollyn, V., Sharma, G. & Biswas, P. (2020, June). Webcam Controlled Robotic Arm for Persons with SSMI, Technology and Disability 32 (3), IOS Press 2020, … disability/tad200264

Citation: Gaze-controlled robotic arm for the speech and motor impaired (2020, July 13) retrieved 29 September 2023 from
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Explore further

New techniques for eye-gaze tracking could change computer interaction


Feedback to editors