July 25, 2018 weblog
Eggs, jokes, and weather report: Alexa's adventure in American Sign Language
Abhishek Singh asks a simple question: If voice is the future of computing what about those who cannot hear? Alexa is all ears for the deaf community thanks to an app prototype that will no doubt draw some interest and inspiration.
After all, tech watching pundits have said that voice technology figures well in our computing futures; as BBC News said, "The past few years have seen a rise in popularity of voice assistants run by Amazon, Google and Apple."
Thanks to Abhishek Singh, Alexa can respond to sign language. Want to add eggs to your shopping list? Need to know if it's raining in Manhattan? No problem, Alexa can answer such questions for those who cannot speak clearly enough or hear.
Singh's ingenious project involved a camera-based system. He had been thinking about it might take for home devices to be designed with deaf users in mind. As a result, he rigged Amazon's Alexa, said the BBC, to respond in text to American Sign Language (ASL).
Singh told YouTube visitors in a video showing his sign language in action that he used deep learning with TensorFlow.js to make Amazon Echo respond to sign language.
Specifically, as the BBC noted, "The developer trained an AI using the machine-learning platform Tensorflow, which involved repeatedly gesturing in front of a webcam to teach the system the basics of sign language."
Once the system was able to respond to his hand movements, he connected it to Google's text-to-speech software to read the corresponding words aloud.
Singh said that at the outset, "As a thought experiment, I used deep learning to make AIexa respond to sign language."
The camera interprets the user's signs. Signs are converted to text and speech. "Alexa, hello," comes out loud and clear. "Hi, there." "Alexa, what is the weather?" "Right now in New York it's 29 degrees Celsius with partly sunny skies. Today's forecast has lots of clouds..." "Alexa, what is five feet in meters?" "Five feet is 1.5 meters." And so on. And what would a session report be without a request for a joke: "Alexa, tell me a joke." I won't repeat it, just because it is lame. On that note (can't blame him) Abhishek said "Alexa, bye."
Question is, could his prototype have any influence on the future of voice assistants? The BBC quoted Singh as saying that there was no reason that Amazon Show "or any of the camera and screen based voice assistants couldn't build this functionality right in."
Fast Company quoted Singh: "In an ideal world I would have built this on the Show directly, but the devices aren't that hackable yet, so wasn't able to find a way to do it."
Mark Wilson, a senior writer at Fast Company, meanwhile, said Singh "will be open-sourcing his own code and sharing the full methodology behind it."
He said Singh hoped other people could build on it , or even just "be inspired to explore this problem space." Wilson had a succinct summing up of how the deaf have been left behind in the gee-whiz race for presenting voice assistants. Wilson said the problem was two-fold.
"These devices never learned to decipher the spoken voices of people with an extreme hearing impairment. At the same time, anything Home or Alexa say in response can't be heard by the user. Adding a screen to display information on a device like the Echo Show might help, but it can only get someone so far if they want to have a natural conversation with a machine."
KnowTechie called it an "elegant" solution and said that "It's awesome to see developers are stepping up for accessibility, but it would be great to see this built-in."
© 2018 Tech Xplore