Adobe's VideoGigaGAN uses AI to make blurry videos sharp and clear

A team of video and AI engineers at Adobe Research has developed an AI application called VideoGigaGAN, that can accept a blurry video and enhance it to make it a much shaper product. The team describes their work and results in an article posted to the arXiv preprint server. They have also posted several examples of the videos that they have enhanced on their project website page.

AI applications have been in the news a lot lately, primarily due to the release of LLMs, such as ChatGPT, that consumers can use to generate a wide variety of output. But AI research has been ongoing in other areas as well, such as creating artificial images and video.

In this new effort, the team at Adobe has created an application that can accept a blurry video sample and, after processing, return the same sample with greatly enhanced sharpness and clarity—also known as upscaling.

It is called VideoGigaGAN—the name comes from its previously demonstrated app, GigaGAN, which generated new photos or improved old ones. GAN stands for generative adversarial network.

As its name implies, the team used a generative adversarial network to teach the system what sharp and clear video looks like (such as individual hairs in eyebrows, rather than a blurry mass) and then added a "flow-guided propagation module" to keep things consistent between video frames.

They also used anti-aliasing techniques to prevent what they describe as "AI weirdness," and high-frequency feature shuttling to handle unexpected declines in video quality.

The result, the team claims, is a system that can upscale video image quality by up to eight times—all without introducing odd coloring, uneven lines or other well-known problems with AI-generated images and video.

They acknowledge that some of the output is entirely artificially generated based on estimates made by the system as it looks to fill out missing imagery. Pores in skin, for example, or lines around the eyes, or even eyelashes, are added to give the resulting video a sharp and clear quality.

The team notes that as of now, the announcement of the system is a demonstration, not of a pending release; thus, it is not clear if Adobe will be releasing it for general use.

© 2024 Science X Network