Levitation, touch and sound – how you'll be able to feel videogames in the future

Despite advances in both virtual and augmented reality technology in the last few years, there's one area that remains neglected: touch. With your VR headset on, you might be able to explore the sights of a vast forest and hear birdsong all around you, but you won't feel the dampness of the moss on a tree trunk or the squelch of leaves underfoot.

And yet touch is an integral part of how we interact with the world. A hug from a loved one can make a terrible day feel better, the feeling of wind whipping past while riding a bike can be exhilarating, even tactile feedback from a button at a pedestrian crossing is reassuring.

"You explore and understand the world through all of your senses, but when you come to a computer, you're really cut off from some of those," says Stephen Brewster, a professor of human-computer interaction at the University of Glasgow, UK. "Thinking of the varied interactions you have with the everyday physical world outside of computers, could we not bring some of that in?"

Bringing elements of touch into our digital lives is exactly what he and other researchers working on haptic—or touch-based—technology are trying to do. Prof. Brewster is part of Levitate, a project which aims to create a prototype of a levitating object that users can reach into and manipulate as easily as if it were digital pixels on a screen, complete with tactile feedback.

Ultrasound

The idea is to use ultrasound—the same technology used in car sensors to stop you hitting the wall when parking—to create three different effects. The first is 'feelable forces' in mid-air, imitating the feeling of touch without there being a physical object present. The second is what's known as parametric sound, where a speaker can emit highly focussed audio that is heard just by one person and not, for example, the person sitting next to them. And the third is what gives the project its name: levitation of small objects.

Now the team is beginning to bring all three aspects together—touch, sound, and levitation—with the aim to eventually to run them off the same ultrasound speaker system.

The work is not without challenges. Prof. Brewster and his colleagues have levitated multiple small polystyrene beads in the shape of a cube that can be rotated in mid-air in response to gestures. Adding this interaction with the cube was a challenge. "You want objects to be able to change shape or deform (when someone pushes them)," he said. "It becomes complex to enable those dynamic movements in response to your gestures."

The sensation of touch comes about when ultrasound waves from different emitters combine at focal points that are moving incredibly quickly, creating the illusion of a solid object in mid-air. The idea is that if you're wearing an AR or VR headset and seeing a virtual object in front of you—or interacting with some combination of physical pixels like the Levitate project's polystyrene beads—you could also feel the object as you interact with it.

Sensitive

Part of the reason our sense of touch has been so far neglected in the digital realm is that it's just harder to recreate touch than an image or a sound. "Our fingers are so sensitive, our skin is so sensitive, it's very difficult to make hardware that's as good as our as our skin is," said Prof. Brewster. "So oftentimes those touch-based interactions are a bit poor."

When you run your finger across your desk, say, the surface of your skin moves up and down, creating waves that travel through the layers of your skin and down to your bone. On their way, they hit mechanical receptors with nerves that fire an electrical signal to your brain.

Dr. Tom Montenegro-Johnson and Dr. James Andrews, applied mathematicians at the University of Birmingham, UK, are modelling this process—up to the point at which the receptor sends a signal to the brain—to help create more realistic haptic feedback.

"If we can recreate the first part of that process as close as possible, with our haptic devices, the rest of it should hopefully take care of itself," said Dr. Montenegro-Johnson.

He's been able to take advantage of a wealth of mathematics that already exists in an entirely separate area of science. "When we started this, we noticed that the skin on humans is, in a sense, very similar to the skin on the Earth, mathematically speaking," he said. "There's an almost one-to-one map between what happens in earthquakes and what happens when you rub your finger lightly over a surface."

The research is part of H-Reality, a project, like Levitate, using ultrasound to create sensations of touch in mid-air. The idea is to use this new model of touch to create the next generation of wearable haptics—a device you wear on your hand that provides feedback on both the shape and texture of digital objects.

Bomb disposal

Dr. Montenegro-Johnson thinks the most widespread application for this kind of technology will be video gaming. But there are other, more niche but perhaps more worthwhile, applications, too. Bomb disposal robots could pass information on to a computer programme that feeds into a wearable haptic device for the person controlling it, for example.

"If you have a robot arm defusing a bomb, or doing some delicate operation, and you're controlling it via a VR interface, we will now be able to give you touch feedback," said Dr. Montenegro-Johnson.

With levitating objects that provide touch feedback, scientists could sit around a model of a protein floating in mid-air and manipulate it as they discuss it, or designers could reach in and make changes to their work in a real 3-D model, not a 2-D computer-simulation of one. This technology could also be added to the devices we already use daily. "You might put your hand over the phone and feel the number of messages," said Prof. Brewster.

As computers of all sizes encroach into our lives ever further, bringing in more of our senses would enrich our interactions with technology. "Rather than just having massive screens and me stabbing a keyboard with my fingers... I can do a lot more than that, as a human," said Prof. Brewster. "Why do my devices not pay any attention?"