Credit: CC0 Public Domain

A report by Sony software engineers reveals the company is developing a highly immersive virtual reality motion system likely to be used in its next generation PlayStation controllers.

The VR controllers would employ finger-tracking technology.

In a paper titled "Evaluation of Machine Learning Techniques for Hand Pose Estimation on Handheld Device with Proximity Sensor," a team from Sony's Interactive Entertainment division state they have developed a controller that accurately replicates the movements of a player's hands and fingers.

Finger tracking applications commonly have relied on cameras that capture finger movement, such as the Oculus Quest. More recently, capacitive proximity sensors that offer more accurate motion recognition are entering the market. But, as Sony researchers Kazuyuki Arimatsu and Hideki Mori note, camera detection is hampered by the presence of physical controllers, and proximity sensor technology, still in its infancy, needs improvement and refinement. Accurately capturing hand movement remained elusive.

Arimatsu and More have taken on the challenge of filling those gaps. They constructed a prototype hand controller with 62 embedded electrodes and recorded training datasets with an optical monitoring system. They captured movements of the hands of 12 subjects, each with varying hand sizes.

"We aim to push the boundary of finger-tracking capability for more intuitive interaction using hands in ," Arimatsu and More said. "Our approach not only detects touching of fingers on a specific electrode but predicts comprehensive finger movement in 3D space utilizing values from all sensors. To achieve that, we evaluated two types of convolutional neural network architectures studied in the computer vision field for pose estimation, and illustrated the suitable architecture for the sensors on the controller."

To achieve realistic and immersive 3D experiences, the researchers said in a report posted last month, they applied multi-dimensional representation "to apply convolutional neural network methods on a capacitive image of the curved surface and two types of network architectures based on recent achievements in the computer vision field were evaluated with our dataset. We also implemented real-time interactive applications using the prototype and demonstrated the possibility of intuitive interaction using fingers in VR applications."

The prototype video shows a user's hand and its digital representation making various gestures such as opening and closing, counting, thumb rotation, spreading fingers, peace sign and outstretched thumb and pinky (symbolizing a phone call). It can recognize a gripping the controller device and lifting or throwing an object.

Last year, Sony licensed advanced haptics for VR controllers. They will be able to simulate sensations of pushing, pulling, grasping and pulsing. The next Sony may well allow game players to not only see their hands but to feel various objects they interact with.

The team said this technology could be applied to areas beyond gaming, such as non-verbal communication applications.

More information: Evaluation of Machine Learning Techniques for Hand Pose Estimation on Handheld Device with Proximity Sensor, CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, dl.acm.org/doi/fullHtml/10.1145/3313831.3376712