Neural network used to create realistic-looking 'photograph' from line drawn face

Neural network used to create realistic-looking ‘photograph’ from line drawn face
Example results of our convolutional sketch inversion models. Our models invert face sketches to synthesize photorealistic face images. Each row shows the sketch inversion / photo synthesis pipeline that transforms a different sketch of the same face to a different image of the same face via a different deep neural network. Each deep neural network layer is represented by the top three principal components of its feature maps. Credit: Yagmur Gucluturk et al., arXiv:1606.03073 [cs.CV]

(Tech Xplore)—A small team of researchers at Radboud University in the Netherlands has used neural networking technology to create a system capable of accepting a sketch of a human face and effectively converting it to one that looks like a photograph. In their paper uploaded to the preprint server arXiv, the team describes their approach, how well it worked and some applications they believe might benefit from its use.

Before the advent of , humans sketched the faces of other humans and over time learned to paint them as well. Such efforts were eventually overshadowed by cameras which allowed anyone to capture a near perfect rendering. That led to efforts by some to do the reverse—to create a sketch of a person's face by looking at a photograph of them. Eventually computer scientists developed software to carry out this feat as well, often with surprising and/or amusing results. Now, the team in the Netherlands has found a way to do the reverse—to create a photo-realistic image of a person's face, using only a sketch of it.

The team first used conventional software to troll the Internet looking for pictures of peoples' faces. They wound up using 200,000 of them (including many of celebrities), which they then used to convert to sketches. Those sketches were then fed to an 11 layer that the team had programmed to learn as it went, to convert the images to photo-realistic images. The results, as can be seen by rendered photos in their paper, are quite remarkable. Of particular note are the skin tones, which the network surmised by studying differences in luminance as well as ethnic characteristics. Such an ability, the team notes, is likely to be quite useful in law enforcement efforts after sketches are made using eyewitness descriptions.

Taking their efforts in another direction, the team used their system on three made by famous painters from the past (Rembrandt, Vincent van Gogh and M.C. Escher), of themselves. Those results have also been presented in their paper, along with reference images for comparison—the researchers describe them as interesting and possibly of interest to those in the fine arts looking for more background information on people from the past who have no photographic record.

Neural network used to create realistic-looking ‘photograph’ from line drawn face
Examples of the synthesized inverse sketches from the LFW dataset. Each distinct column shows examples from different sketch styles models, i.e. line sketch model (column 1), grayscale sketch model (column 2) and colour sketch model (column 3). First image in each column is the ground truth, the second image is the generated sketch and the third one is the synthesized inverse sketch. Credit: Yagmur Gucluturk et al., arXiv:1606.03073 [cs.CV]
More information: Convolutional Sketch Inversion, arXiv:1606.03073 [cs.CV]

In this paper, we use deep neural networks for inverting face sketches to synthesize photorealistic face images. We first construct a semi-simulated dataset containing a very large number of computer-generated face sketches with different styles and corresponding face images by expanding existing unconstrained face data sets. We then train models achieving state-of-the-art results on both computer-generated sketches and hand-drawn sketches by leveraging recent advances in deep learning such as batch normalization, deep residual learning, perceptual losses and stochastic optimization in combination with our new dataset. We finally demonstrate potential applications of our models in fine arts and forensic arts. In contrast to existing patch-based approaches, our deep-neural-network-based approach can be used for synthesizing photorealistic face images by inverting face sketches in the wild.

Journal information: arXiv

© 2016 Tech Xplore

Citation: Neural network used to create realistic-looking 'photograph' from line drawn face (2016, June 16) retrieved 1 December 2022 from
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Explore further

Neural networks allow classic painting styles to be applied to modern video


Feedback to editors