December 14, 2018 feature
A new technique for synthesizing motion-blurred images
Researchers at Google have recently developed a new technique for synthesizing a motion blurred image, using a pair of un-blurred images captured in succession. In their paper, pre-published on arXiv, the researchers outlined their approach and evaluated it against several baseline methods.
Motion blur naturally occurs when the objects in a scene or the camera itself are moving as an image is being taken. This results in the moving object/s or the entire image appearing blurred. In some cases, motion blur might be used to indicate the speed of a photographed subject or to separate it from the background.
"Motion blur is a valuable cue in the context of image understanding," Tim Brooks and Jonathan Barron, the researchers who carried out the study, wrote in their paper. "Given a single image containing motion blur, one can estimate the relative direction and magnitude of scene motion that resulted in the observed blur. This motion estimate may be semantically meaningful, or may be used by a de-blurring algorithm to synthesize a sharp image."
Recent research has investigated the use of deep learning algorithms to remove unwanted motion blur from images or for inferring the motion dynamics of a given scene. To train these algorithms, however, researchers need a substantial amount of data, which is typically generated by synthetically blurring sharp images. Ultimately, the extent to which a deep learning algorithm can effectively remove motion blur in real images greatly depends on the realism of the synthetic data used to train it.
"In this paper, we treat the inverse of this well-studied blur estimation/removal task as a first class problem," Brooks and Barron wrote in their paper. "We present a fast and effective way to synthesize the training data necessary to train a motion de-blurring algorithm, and we quantitatively demonstrate that our technique generalizes from our synthetic training data to real motion-blurred imagery."
The neural network architecture devised by Brooks and Barron includes a novel "line prediction" layer, which teaches a system to regress from image pairs of consecutively taken images to a motion blurred image that spans the capture time of these two input images. Their model requires a vast amount of training data, so the researchers designed and executed a strategy that uses frame interpolation techniques to generate a large synthetic dataset of motion blurred images, along with their respective inputs.
Brooks and Barron also captured a high quality test set of real motion blurred images synthesized from slow motion videos and then used this to evaluate their model against baseline techniques. Their model achieved very promising results, outperforming existing approaches in both accuracy and speed.
"Our approach is fast, accurate, and uses readily available imagery from videos or 'bursts' as input, and so provides a path for enabling motion blur manipulation in consumer photography applications, and for synthesizing the realistic training data needed by deblurring or motion estimation algorithms," the researchers wrote in their paper.
While experienced photographers and cinematographers often use motion blur as an artistic effect, producing effective motion-blurred photographs can be very challenging. In most cases, these images are the product of a long trial and error process, requiring advanced skills and equipment.
Due to the difficulties in achieving quality motion blur effects, most consumer cameras are designed to take images with as little motion blur as possible. This means that amateur photographers have very little space to experiment with motion blur in their images.
"By allowing motion blurred images to be synthesized from the conventional un-blurred images that are captured by standard consumer cameras, our technique allows non-experts to create motion blurred images in a post-capture setting," the researchers explained in their paper.
Ultimately, the approach devised by Brooks and Barron could have a number of interesting applications. For instance, it could make artistic motion blur accessible to casual photographers, while also generating more realistic motion blurred images to train deep learning algorithms.
© 2018 Science X Network