The virtual made real: New technology for the media of the future

Moving images in the newspaper as in Harry Potter, or digital weather forecasts in the Bern dialect? New media technology is transforming journalism. The Media Technology Center at ETH Zurich supports media companies on their path to the future.

In the magical world of boy wizard Harry Potter, all manner of things are possible—for example, in the Daily Prophet, the newspaper for witches and wizards, the illustrated figures move just as they do in real life.

Naturally, you won't find this in the real world—yet. But modern media technology is getting ever closer to it. In the Media Technology Center (MTC) at ETH Zurich, for example, students and researchers are working on an "augmented paper", which would allow anyone with a pair of special glasses—known as augmented reality glasses or AR glasses—to actually see moving pictures on the page, as in Harry Potter. "It is quite possible that these glasses will soon be small enough for us to wear all day," says Severin Klingler, Managing Director of MTC. "Then we'll see how digital and real content can be blended together in a 'augmented paper'."

Harry Potter starts school—a video for the paper

The "augmented paper" that ETH researchers are currently developing combines the advantages of a traditional newspaper made of paper with the advantages of digital content. "You hold the paper in your hands as normal, but when you read it using the AR glasses the pictures and words change," explains Klingler.

For example, in a picture below the headline "Start of school for witches and wizards", you see a steam train emerging. In future, this technology could be used to embed videos of real events in the paper, such as the annual burning of the Böögg at the Sechseläuten festival in Zurich. "What's new about our technology is how the moving pictures are still displayed correctly even when the pages of the paper bend as they are turned over," says Klingler.

MTC is pursuing a very practical goal with its project for AR glasses in Zurich city centre. Anyone who looks at Paradeplatz through AR glasses will see local information such as tram departure times, advertising posters for the surrounding shops—or newspaper articles about the two major banks located there. The focus is on questions of practical legibility: Where and when is an advertising poster displayed? Where are the articles displayed? Which surfaces are suitable for the displays?

A dinosaur strolling through Zurich

"Above all, MTC is concerned with how these new types of digital content can be generated for the media in the simplest way possible," says Klingler. Using MTC's tool, it is easy to augment Paradeplatz with virtual characters—turning it into a zoo or an underwater world, for example. "Everything that someone creates in virtual reality appears automatically on the real Paradeplatz," explains Klingler. "Our tool means users no longer have to grapple with the difficult question of how to get their virtual content into the right real place. They can simply concentrate on the design."

One of the technical challenges, for which there is still no simple solution, is "occlusion handling": unlike a film made for the cinema, which is edited in a studio, virtual characters moving through a real situation in real time often fail to disappear from view immediately when a user turns a corner in the street. This disrupts the credibility of the visual experience.

The ETH researchers have devised an approach that makes this work better—for example, a virtual dinosaur wandering around near the tram stop in Stampfenbachplatz now disappears from view immediately as in real life as it turns the corner by the canton of Zurich's media center. The ETH researchers used a 3-D model of the city of Zurich to achieve this.

Local weather in local dialects

Besides virtual moving pictures, voice assistants are likely to become increasingly important in our daily use of media. The media technology trend indicates that media content such as videos, radio news or texts will be retrieved less by hand—through clicking, typing or swiping—and more frequently by voice. We are already familiar with talking intelligent assistants from fitness apps or traffic forecasts on our smartphones.

"But none of the assistants available today speak or understand Swiss dialects, which makes communication with them rather unnatural for Swiss people," says Klingler. This is why MTC is developing a voice assistant that speaks Swiss German. It has been speaking fluent "Bärndütsch", the dialect of Bern, since 2019, and is currently being expanded to include other dialects.

As is characteristic of the work of MTC, the researchers are developing the Swiss German assistants in partnership with a Swiss media company. In this case, the partner is the main Swiss radio and television network, SRF. SRF publishes large amounts of content in local dialects on its channels. Technology that translates High German to Swiss German or that can announce the local news and weather forecast in the local dialect is therefore highly practical and gives the automatically spoken text a regional authenticity.

Support for digital transformation

MTC's development projects are generally implemented in one to two years and offer practical benefits to editorial offices and media companies. The aim is to support Swiss media companies in their digital transformation. This applies to both the generation of journalistic content and the production of various media formats and methods that bring digital content to users and personalize media consumption.

The Media Technology Center is part of the ETH media technology initiative. The center was established in 2019 with partners from the media industry. It is supported by the media companies Ringier, TX Group (previously Tamedia), SRG SSR, NZZ, Verband Schweizer Medien, the association of employers and entrepreneurs in the Swiss printing industry viscom and the ETH Zurich Foundation. All partners—like ETH Zurich—have contributed financially to make the MTC possible. The center is still open to expand its partnerships. The scientific head of the center is ETH computer science professor Markus Gross, who is joined by experts from the business and scientific sectors.

Open for media project ideas

"We are open to any project ideas from media companies and editorial offices," says Klingler. MTC began its work in March 2019 and comprises eight researchers, supported by students and doctoral students, in the fields of computer vision, computer graphics, natural speech processing, recommendation systems, user modeling and machine learning. Ryan Cotterell has also been working in Zurich since February 2020. The computer science professor was appointed as part of the media technology initiative, and combines linguistics, automated speech processing and artificial intelligence.