June 16, 2016 report
Neural network used to create realistic-looking 'photograph' from line drawn face
(Tech Xplore)—A small team of researchers at Radboud University in the Netherlands has used neural networking technology to create a system capable of accepting a sketch of a human face and effectively converting it to one that looks like a photograph. In their paper uploaded to the preprint server arXiv, the team describes their approach, how well it worked and some applications they believe might benefit from its use.
Before the advent of modern technology, humans sketched the faces of other humans and over time learned to paint them as well. Such efforts were eventually overshadowed by cameras which allowed anyone to capture a near perfect rendering. That led to efforts by some to do the reverse—to create a sketch of a person's face by looking at a photograph of them. Eventually computer scientists developed software to carry out this feat as well, often with surprising and/or amusing results. Now, the team in the Netherlands has found a way to do the reverse—to create a photo-realistic image of a person's face, using only a sketch of it.
The team first used conventional software to troll the Internet looking for pictures of peoples' faces. They wound up using 200,000 of them (including many of celebrities), which they then used conventional software to convert to sketches. Those sketches were then fed to an 11 layer deep neural network that the team had programmed to learn as it went, to convert the images to photo-realistic images. The results, as can be seen by rendered photos in their paper, are quite remarkable. Of particular note are the skin tones, which the network surmised by studying differences in luminance as well as ethnic characteristics. Such an ability, the team notes, is likely to be quite useful in law enforcement efforts after sketches are made using eyewitness descriptions.
Taking their efforts in another direction, the team used their system on three sketches made by famous painters from the past (Rembrandt, Vincent van Gogh and M.C. Escher), of themselves. Those results have also been presented in their paper, along with reference images for comparison—the researchers describe them as interesting and possibly of interest to those in the fine arts looking for more background information on people from the past who have no photographic record.
More information: Convolutional Sketch Inversion, arXiv:1606.03073 [cs.CV] arxiv.org/abs/1606.03073
Abstract
In this paper, we use deep neural networks for inverting face sketches to synthesize photorealistic face images. We first construct a semi-simulated dataset containing a very large number of computer-generated face sketches with different styles and corresponding face images by expanding existing unconstrained face data sets. We then train models achieving state-of-the-art results on both computer-generated sketches and hand-drawn sketches by leveraging recent advances in deep learning such as batch normalization, deep residual learning, perceptual losses and stochastic optimization in combination with our new dataset. We finally demonstrate potential applications of our models in fine arts and forensic arts. In contrast to existing patch-based approaches, our deep-neural-network-based approach can be used for synthesizing photorealistic face images by inverting face sketches in the wild.
© 2016 Tech Xplore
