Eggs, jokes, and weather report: Alexa's adventure in American Sign Language

Credit: CC0 Public Domain

Abhishek Singh asks a simple question: If voice is the future of computing what about those who cannot hear? Alexa is all ears for the deaf community thanks to an app prototype that will no doubt draw some interest and inspiration.

After all, tech watching pundits have said that voice technology figures well in our computing futures; as BBC News said, "The past few years have seen a rise in popularity of voice assistants run by Amazon, Google and Apple."

Thanks to Abhishek Singh, Alexa can respond to . Want to add eggs to your shopping list? Need to know if it's raining in Manhattan? No problem, Alexa can answer such questions for those who cannot speak clearly enough or hear.

Singh's ingenious project involved a camera-based system. He had been thinking about it might take for home devices to be designed with deaf users in mind. As a result, he rigged Amazon's Alexa, said the BBC, to respond in text to American Sign Language (ASL).

Singh told YouTube visitors in a video showing his sign language in action that he used deep learning with TensorFlow.js to make Amazon Echo respond to sign language.

Specifically, as the BBC noted, "The developer trained an AI using the machine-learning platform Tensorflow, which involved repeatedly gesturing in front of a webcam to teach the system the basics of sign language."

Once the system was able to respond to his hand movements, he connected it to Google's text-to-speech software to read the corresponding words aloud.

Singh said that at the outset, "As a thought experiment, I used to make AIexa respond to sign language."

The camera interprets the user's signs. Signs are converted to text and speech. "Alexa, hello," comes out loud and clear. "Hi, there." "Alexa, what is the weather?" "Right now in New York it's 29 degrees Celsius with partly sunny skies. Today's forecast has lots of clouds..." "Alexa, what is five feet in meters?" "Five feet is 1.5 meters." And so on. And what would a session report be without a request for a joke: "Alexa, tell me a joke." I won't repeat it, just because it is lame. On that note (can't blame him) Abhishek said "Alexa, bye."

Question is, could his prototype have any influence on the future of voice assistants? The BBC quoted Singh as saying that there was no reason that Amazon Show "or any of the camera and screen based voice assistants couldn't build this functionality right in."

Fast Company quoted Singh: "In an ideal world I would have built this on the Show directly, but the devices aren't that hackable yet, so wasn't able to find a way to do it."

Mark Wilson, a senior writer at Fast Company, meanwhile, said Singh "will be open-sourcing his own code and sharing the full methodology behind it."

He said Singh hoped other people could build on it , or even just "be inspired to explore this problem space." Wilson had a succinct summing up of how the deaf have been left behind in the gee-whiz race for presenting assistants. Wilson said the problem was two-fold.

"These devices never learned to decipher the spoken voices of people with an extreme hearing impairment. At the same time, anything Home or Alexa say in response can't be heard by the user. Adding a screen to display information on a device like the Echo Show might help, but it can only get someone so far if they want to have a natural conversation with a machine."

KnowTechie called it an "elegant" solution and said that "It's awesome to see developers are stepping up for accessibility, but it would be great to see this built-in."

Explore further

Bonjour, Alexa: Amazon digital assistant heads to France

© 2018 Tech Xplore

Citation: Eggs, jokes, and weather report: Alexa's adventure in American Sign Language (2018, July 25) retrieved 19 August 2022 from
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Feedback to editors