Algorithm eliminates blurred images caused by shaky footage
Duke University computer engineers have designed algorithms capable of sharpening video blurred by a shaky camera. Newly integrated into Adobe's After Effects video editing software, the solution is bringing relief to tripod-less videographers everywhere.
"When I came to Duke, I wanted to try to find a different solution to a classic problem," said Mauricio Delbracio, the former Duke postdoc who worked on the project with Guillermo Sapiro, the Edmund T. Pratt, Jr. School Professor of Electrical and Computer Engineering at Duke. "I eventually landed on video blur because no simple algorithm existed to tackle this very important problem."
Part of the reason for Delbracio's choice was that other algorithms created to address blurriness did not work very well. The standard approach tried to figure out what caused the image degradation and then invert it.
"We showed mathematically that this process is a waste of time because it doesn't really help you solve the actual problem," said Sapiro. "Thanks to Mauricio's brilliance and lots of discussion, we found a new way to look at the problem."
Sapiro and Delbracio used the very shaking that caused the blur in the first place to find a shortcut for removing it. Because videos take many images in rapid succession, the duo decided to try to use the surrounding frames to sharpen the problematic image. And because a shaky hand causes random movements, each frame is degraded differently.
The researchers wrote an algorithm that analyzes and scores the original frame on sharpness. It then does the same for each region of nearby frames and compares them to the original. By choosing the best available options and compiling them into one image, a new and improved "Frankenstein frame" is created to replace the blurred original.
"Each image has something good and something bad, so we simply tried to merge all of the good parts," explained Sapiro. "We discovered that once you realize all the frames are degraded differently, it becomes very easy to do this mathematically."
The simplicity of the calculations means that the corrections can be made in a matter of seconds, making it a perfect addition to any video-editing software.
"Most algorithms need several minutes to sharpen a single frame," said Delbracio, now an assistant professor at Universidad de la República in Uruguay. "That's totally unacceptable for video software. You need to be under a second."
A prominent figure in the field for more than a decade and having licensed his work before, Sapiro has several contacts at various video analysis companies. After publishing his lab's work in IEEE Transactions on Computational Imaging in June 2016, he reached out Adobe to see if they might be interested in integrating the new method into one of their video editing platforms.
They were.
Delbracio spent about a month working with colleagues at Adobe to make sure his algorithm worked in their environment. Writing a computer code for a scientific paper is one thing; integrating it into a platform in an intuitive way so that users can help shape the results is quite another.
The collaboration was successful, however, and the feature was released in the Spring 2017 version of Adobe After Effects. The addition most notably allows users to take shaky footage and make it stable, all while avoiding blurred lines.
"They already knew geometrically how to take the shake out of unstable footage," said Delbracio. "But stabilizing the field introduces blur on top of the image, so it's important to do both to recover usable video."
