Tricking the brain: New dimensions of haptics in virtual reality

How can virtual reality (VR) be experienced haptically, i.e., through the sense of touch? This is one of the fundamental questions that modern VR research is investigating.

Computer scientist André Zenner, who is based in Saarbrücken, Germany, has come a significant step closer to answering this question in his doctoral thesis—by inventing new devices and developing software-based techniques inspired by human perception. He has now been awarded the "Best Dissertation Award" at the world's leading VR conference.

The award-winning work is about how physical props (technical term: "proxies") can be used to make objects in virtual environments tangible.

"Of course, you can't have a proxy for every virtual object, then the approach wouldn't be scalable. In my dissertation, I, therefore, thought about what devices could look like that could be used to simulate the physical properties of several different virtual objects as effectively as possible," explains Zenner, who completed his doctorate at the Saarbrücken Graduate School of Computer Science at Saarland University and is now conducting research at Saarland University and the German Research Center for Artificial Intelligence.

This resulted in the prototypes for two special VR controllers, "Shifty" and "Drag:on." VR controllers are devices that can be held in the user's hand to control or manipulate objects in virtual reality using tracking technology.

"Shifty" is a tubular controller in which a movable weight is installed. The weight can be moved along the lengthwise axis by a motor, changing the center of gravity and inertia of the rod.

"In combination with corresponding visualizations in virtual reality, Shifty can be used to create the illusion that a virtual object is getting longer or heavier," explains Zenner. In experiments, he was able to prove that objects are perceived as lighter or smaller when the weight is close to the user's hand and that, coupled with the corresponding visual input; they are perceived as longer and heavier the further the weight in the rod moves away from the user.

"This is mainly due to changes in the inertia of the controller, as the overall weight does not change," explains Zenner. The research and development department of gaming giant Sony is already experimenting with this concept and cites Zenner's work in the development of new VR controllers.

The second controller, "Drag:on," consists of two flamenco fans that can be unfolded using servomotors, thus increasing the air resistance of the controller. This means that the further the fans are unfolded, the more force the user has to exert to move the controller through the air.

"Coupled with the right visual stimuli, Drag:on can be used to create the impression that the user is holding a small shovel or a large paddle, for example, or that they are pushing a heavy trolley or are twisting a knob that is difficult to turn," explains Zenner.

Both controllers are basic research and so-called "proof of concepts." This means that the prototypes can be used to show in user experiments that different controller states can improve the perception of different VR objects. Still, specific products using this technology are not yet available on the market.

With the controllers, the Saarbrücken-based computer scientist first addressed the so-called 'similarity problem." The aim here is to ensure that virtual and real objects feel as similar as possible. In the second part of his work, he dealt with the so-called "colocation problem," i.e., the question of how the proxy can be spatially located in real life where the user sees it in virtual reality.

This is particularly challenging as the controllers act as proxies for different virtual objects. Consequently, the user must be given the illusion that they are reaching for various objects, although in reality, they will always grasp the same proxy.

To achieve this, the researcher made use of the already established method of "hand redirection." As the name suggests, this involves redirecting the movement of the hand in virtual reality so that the user thinks they are reaching to the left, for example, even though they are actually stretching their hand forward.

"We conducted experiments to investigate the point at which users realize that their hand has been redirected. Our results showed that this point was reached quickly, so we thought about how we could better conceal the hand redirection," says Zenner.

The solution: he tricked the brain by only redirecting the hand when the brain was blind to visual changes—namely during blinking. Together with a student under his supervision, he developed the appropriate software and used the eye trackers built into many VR headsets.

In control studies, the team was then able to show that their new controllers, in combination with hand redirection algorithms, led to more convincing VR perceptions than previously possible.