VR system mixes physical and virtual worlds to drive home climate urgency

An iconic red shuttle bus ferries commuters and visitors along the winding streets of New York City's Roosevelt Island. But this isn't a typical sightseeing tour.

Passengers all don virtual reality headsets for an eye-opening experience—a cutting-edge blend of the physical and digital worlds, designed to engage communities in new ways through the Communal eXtended-Reality (CXR) system.

Passengers were transported into a realm where virtual environments seamlessly merged with their real surroundings. Guided by audio narration, they encountered nine striking scenes depicting the escalating impacts of climate change, with a particular focus on rising floodwaters.

"In the fragmented media landscape we live in, and with differences in regional issues, the shared physical experience is a very unique way to bring people together," said Wendy Ju, associate professor at the Jacobs Technion-Cornell Institute at Cornell Tech, whose research group created CXR.

Ju and her team presented the CXR system at the July 2024 ACM SIGCHI Conference on Designing Interactive Systems, where it earned an honorable mention. Co-authors included Cornell Tech postdoctoral associate and project lead Sharon Yavo-Ayalon, along with doctoral students Adam Yuzhen Zhang and Fanju Bu; Cooper Murr; and researchers from New York University and the University of Massachusetts, Amherst.

Projections suggest that a 100-year flood could submerge most of Roosevelt Island. Despite these alarming forecasts, research indicates that such a catastrophe doesn't feel real to many in the public.

This disconnect between knowledge and urgency is precisely what the CXR system aims to address. The technology—which synchronizes real-world movement with virtual overlays—provides a fully immersive, shared extended reality (XR) experience designed to foster a unified understanding of pressing challenges, such as climate change.

This integration allows participants to experience a shared reality while physically traveling through their environment, making the experience both communal and deeply rooted in place.

The deployment of the CXR system on Roosevelt Island proved to be more than just an educational exercise. After seeing "worst-case" flooding scenarios, scenes of Superstorm Sandy, and future sea-level rise forecasts, participants reported strong emotional responses, with many expressing increased concern about climate change and a desire to take action. One participant was even moved to tears upon seeing a simulated flood reach the steps of her school.

These types of reactions were not surprising, Ju said.

"In the planning and development phase, we were warned about scaring people about flooding and climate change because people can become hopeless," she said. "We tried very hard to employ humor and emphasize possible interventions to mitigate this. We did find that people got worried, but we also noted that the worry seemed to galvanize people to act, which is a good outcome from our perspective."

Potential applications of the CXR system extend far beyond its initial deployment, Ju said. CXR can play a crucial role in urban planning by engaging communities in discussions about new developments. By providing a clear and immersive presentation of proposed changes, it helps to reduce misunderstandings and conflicts, fostering a more informed and collaborative approach to future planning.

"Immersive technologies can bring future challenges into focus," Ju said, "which is important for generating the social will to address and prepare for things to come."