MeInGame: A deep learning method to create videogame characters that look like real people

MeInGame: A deep learning method to create videogame characters that look like real people
First row: input portraits. Second row: in-game characters generated by the researchers’ method. Their technique is robust to lighting changes, shadows, and occlusions, and can faithfully restore personalized details like skin tone, makeup, and wrinkles. Credit: Lin, Yuan & Zou.

In recent years, videogame developers and computer scientists have been trying to devise techniques that can make gaming experiences increasingly immersive, engaging and realistic. These include methods to automatically create videogame characters inspired by real people.

Most existing methods to create and customize videogame characters require players to adjust the features of their 's face manually, in order to recreate their own face or the faces of other people. More recently, some developers have tried to develop methods that can automatically customize a character's face by analyzing images of real people's faces. However, these methods are not always effective and do not always reproduce the faces they analyze in realistic ways.

Researchers at Netease Fuxi AI Lab and University of Michigan have recently created MeInGame, a that can automatically generate character faces by analyzing a single portrait of a person's face. This technique, presented in a paper pre-published on arXiv, can be easily integrated into most existing 3D videogames.

"We propose an automatic character face creation method that predicts both facial shape and texture from a single portrait, and can be integrated into most existing 3D games," Jiangke Lin, Yi Yuan and Zhengxia Zou, the three researchers who carried out the study, wrote in their paper.

Some of the automatic character customization systems presented in previous works are based on computational techniques known as 3D morphable face models (3DMMs). While some of these methods have been found to reproduce a person's facial features with good levels of accuracy, the way in which they represent geometrical properties and spatial relations (i.e., topology) often differs from the meshes utilized in most 3D videogames.

MeInGame: A deep learning method to create videogame characters that look like real people
Comparison of the technique devised by the researchers with other methods used in games, namely: A Dream of Jianghu, Loomie, Justice (Shi et al. 2020), ZEPETO. In the last column: the results attained by a 3DMM-based method (Deng et al. 2019). Credit: Lin, Yuan & Zou.

In order for 3DMMs to reproduce the texture of a person's face reliably, they typically need to be trained on large datasets of images and on related texture data. Compiling these datasets can be fairly time consuming. Moreover, these datasets do not always contain real images collected in the wild, which can prevent models trained on them from performing consistently well when presented with new data. To overcome this limitation, Lin, Yuan and Zou trained their technique on a dataset of images captured in the wild.

"Given an input face photo, we first reconstruct a 3D face based on a 3D morphable face model (3DMM) and (CNNs), then transfer the shape of the 3D face to the template mesh," the researchers explained in their paper. "The proposed network takes the face photo and the unwrapped coarse UV texture map as input, then predicts lighting coefficients and refined texture maps."

Lin, Yuan and Zou evaluated their deep learning technique in a series of experiments, comparing the quality of the game characters it generated with that of character faces produced by other existing state-of-the-art methods for automatic character customization. Their method performed remarkably well, generating character that closely resembled those in input images.

"The proposed method does not only produce detailed and vivid game characters similar to the input portrait, but it can also eliminate the influence of lighting and occlusions," the researchers wrote in their paper. "Experiments show that our method outperforms state-of-the-art methods used in games."

In the future, the character face generation method devised by this team of researchers could be integrated within a number of 3D videogames, enabling the automatic creation of characters that closely resemble real people. The MeInGame model's code and the dataset used to train it were published online and can be accessed by game developers worldwide at: github.com/FuxiCV/ MeInGame .


Explore further

A system that automatically generates comic books from movies and other videos

More information: MeInGame: Create a game character face from a single portrait. arXiv:2102.02371 [cs.CV]. arxiv.org/abs/2102.02371

© 2021 Science X Network

Citation: MeInGame: A deep learning method to create videogame characters that look like real people (2021, March 4) retrieved 17 April 2021 from https://techxplore.com/news/2021-03-meingame-deep-method-videogame-characters.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.
151 shares

Feedback to editors

User comments