A system that augments mixed reality visualizations using smartphones or tablets

Mixed reality (MR) and Augmented Reality (AR) technologies merge the real world with computer-generated elements, allowing users to interact with their surroundings in more engaging ways. In recent years, these technologies have enhanced education and specialized training in numerous fields, helping trainees to test their skills or make better sense of abstract concepts and data.

Researchers at University of Calgary have been trying to develop interfaces and systems that could enhanced MR visualizations. In a paper set to be presented at CHI 2023 LBW, they introduced HoloTouch, a system that can augment mixed reality graphics and charts using smartphones as physical proxies.

"To me, this paper was inspired for the most part by a work that I published during my final undergraduate year," Neil Chulpongsatorn, one of the researchers who carried out the study, told Tech Xplore "They both originated from my interest in mixed reality interactions for data representations."

In some of their previous studies, Chulpongsatorn and his colleagues explored how they could augment data visualizations using virtual components showed via head-mounted displays (HMDs), such as Microsoft HoloLens. Other teams also proposed similar solutions, such as physically dynamic bar charts.

While some of their previous efforts yielded interesting results, the researchers realized that current MR and AR headsets did not support such dynamic and enhanced data visualizations. In their recent study, they thus tried to circumvent these limitations using existing and widely used devices.

"Rather than making physical representations interactive, we pivoted to making virtual representation tangible instead," Chulpongsatorn said. "Works like MARVIS presented this idea at a high level and demonstrated its potential. Our recent work digs further down to present the foundation of these interfaces as an interaction space for visualization designers to use in the future, as well as evaluating the effectiveness of these interaction models."

In their recent study, Chulpongsatorn and his colleagues introduced HoloTouch, a system that combines HMDs and the touchscreens on smartphones or tablets to enhance MR visualizations. Essentially, HoloTouch allows users to "touch" and manipulate holograms that they are seeing on HMDs through their smartphone devices.

"Our system addresses some of the challenges of current MR visualizations, such as unreliable tracking, low visual resolution, and imprecise input," Chulpongsatorn explained. "The novelty of our work comes from the design space presented in the paper, and not so much the prototype. Basically, we thought a lot about how mobile phones can interact with holograms (specifically of data visualizations), compiled a reference document, and showcased some examples."

As part of their recent study, the researchers demonstrated six different ways to interact with graphs and other data visualizations using a prototype of their system. These initial tests were highly promising, highlighting the promise of their solution for augmenting data visualizations.

"The results of preliminary evaluations show that people generally like exploring the visualization with the proxy more than without it," Chulpongsatorn said. "Particularly, precision and tangibility stood out as strong advantages of the proxy. This suggests that physical interactions with holograms is beneficial to data analysis task, although more work will have to be done before we can draw stronger conclusions."

In the future, the new system proposed by this team of researchers could be used by designers and content creators to present data to students, professionals, or the public in more immersive ways. Meanwhile, Chulpongsatorn and his colleagues hope to improve their system further, adding new features and interaction styles.

"As this is a late breaking work, there is still a lot to be done," Chulpongsatorn added. "For one, the space currently has breadth but not much depth. There are also other interactions that we still need to explore. We talked about some of this towards the end of the paper. More interactions also mean more evaluation to be done, as that is one of our core contributions with this work."

© 2023 Science X Network