Lip-reading software helps users of all abilities to send secure messages

Lip-reading software helps users of all abilities to send secure messages
Professor Arif, center, with graduate students Laxmi Pandey, left, and Gulnar Rakhmetulla, works on project that benefit computer users of all abilities. Credit: UC Merced

A computer science lab focused on making human-computer interaction easier for people of all abilities has developed a digital lip-reader complete with its own repair system so the software can continue learning from its user.

LipType, a new invention from professor Ahmed Sabbir Arif and his lab, the Human-Computer Interaction Group, lets people send texts or emails on their computers and and have contact-free interactions with public devices such as ATMs or other kiosks, without speaking aloud.

There are other lip-readers, but they are not widely used because they are slow and often faulty, Arif said.

"There are a lot of errors in talk-to-text, especially in noisy places, or for people with speech impairments or those who aren't native speakers," he said. "But LipType works for anyone. People might need to send a private message while in a , or in a meeting, and with LipType, they could just 'say' the words without making a sound."

He and his students added various filters for different lighting conditions and a mistake-corrector based on different language models and they found that LipType was significantly faster and there were fewer errors than with other lip-readers.

To go along with the , Arif's lab conducted a social study to see if people liked and would use such a technology. They reached out to students and people in the community, including people with disabilities, and conducted an online survey. People who tried it in various tests overwhelmingly say they would use it.

How the software was developed. Credit: Arif lab

"People with impairments are often concerned about standing out," Arif said. "This is one way to increase access to mobile and other devices for them in a way that they won't draw attention to themselves."

The social study found that people are willing to use silent speech in public places, even when it is not as accurate as other methods. They feel the software preserves their privacy and security and allows them to do what they need to do without disturbing others around them.

Computer Science and Engineering graduate student Laxmi Pandey, who works with Arif, said she is excited about the results of the tests.

"LipType performed 58 percent faster and 53 percent more accurately than other state-of-the-art models in various real-world settings, including poor lighting and busy market situations," Pandey said. "The success of LipType makes me believe that it can revolutionize our interaction with systems and with other human beings as well."

Credit: University of California

She and Arif have written papers on the social study and LipType. Both have been accepted for publication and presentation at the 2021 Association for Computing Machinery's Special Interest Group on Computer–Human Interaction ACM SIGCHI Conference on Human Factors in Computing Systems, the premier international conference on .

LipType has a number of other applications, as well.

"We were thinking about surveillance situations," Arif said. "This could be very useful for law enforcement. LipType could be used for closed captioning. We're also looking at interfaces for cars. "We have a design-for-all philosophy," Arif said.

Citation: Lip-reading software helps users of all abilities to send secure messages (2021, March 19) retrieved 28 March 2024 from https://techxplore.com/news/2021-03-lip-reading-software-users-abilities-messages.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Explore further

Research brings tech tutorials to people with visual impairments

10 shares

Feedback to editors