February 5, 2019 weblog
Android phone becomes tool for conversations live between deaf and hearing people
Customer: "Hello." Barista: "Hello. What can I get you?" Customer: "I would like tea." For many, this is the most mundane, forgettable exchange between two humans. For those who know limitations confronting the deaf and hard of hearing, the exchange is quite something out of the ordinary.
Dimitri Kanevsky research scientist and engineer Chet Gnegy have worked together to bring out the Android app, Live Transcribe. This app takes speech and turns it into real-time captions using the phone's microphone.
You can see words appear on your Android phone as they're spoken. You join the conversation by typing your response on the screen, via a type-back keyboard.
The hearing populations and the deaf and hard of hearing populations can have conversations easily. The conduit is simply the app on an Android phone.
"We can now do things that weren't even remotely possible a few years ago, like jump into conversations at the dinner table or casually join in when the opportunity arises," said Christian Vogler, Professor and Researcher, Gallaudet University.
The app's creators have taken into account that the app could be enjoyed in more languages than English-only. Potential users will have their choice from over 70 languages and dialects; they can expect accurate captions in the language being spoken.
Wired described the app as enabling "speech-to-text intelligence" where the hard of hearing or deaf can receive "text representations of spoken conversations as they're happening."
A video was posted about this on Monday. "Imagine a world where people who are deaf or hard of hearing can have everyday in-person conversations, in real time."
What puts this show on the road? Kanevesky, deaf since childhood, has worked on speech recognition and communications technology for more than a minute—how about for the last 30 years.
The Live Transcribe team started out with a question/challenge: What if we used cloud-based automatic speech recognition to display spoken words on a screen?
They built a prototype. In a blog, Brian Kernler said Googlers across offices from Mountain View to Taipei got involved. Also, Gallaudet University, the school for the deaf and hard of hearing, pitched in, to check that Live Transcribe was going to be of help for daily use. The school helped with testing and improvements. Gallaudet is lauded as a university for the deaf and hard of hearing, an ideal choice for feedback. The school's programs and services are specifically designed to accommodate deaf and hard of hearing students.
"On the technical side, the app relies on cloud ASR to provide greater accuracy combined with an on-device neural network-based speech detector," said SiliconANGLE. The Google Play product notes the following features of the app: (1) the user can reply without speaking by typing out responses (2) there are optional haptic feedback signals you when somebody starts or resumes speaking.
"Live Transcribe will gradually rollout in a limited beta to users worldwide via the Play Store and pre-installed on Pixel 3 devices. Sign up here to be notified when it's more widely available." (Wired: Live Transcribe will be installed on Google's Pixel 3 smartphones, but otherwise is only available via a beta now.)
Privacy? The site said conversation transcripts aren't stored on servers; they stay secure on your device.
Aatif Sumar, Android Police: "For an app that's supposed to be in beta, it works impressively well. I had conversations around the phone, and the app was able to pick up most of what was said. Even when the transcription missed a word or two, you could get the gist of the conversation."
Sumar wrote about that haptic feature, too. "If the conversation stops for a while and then resumes, the phone can vibrate to alert you."
An interesting observation from Lauren Goode in Wired: Live Transcribe is "less about helping a user navigate the phone itself and more about people using phones to navigate the world that exists around us when we're not buried in our phones."
© 2019 Science X Network