About faces: Geometric style of portrait artwork

A team of computer scientists at the Interdisciplinary Center in Israel have developed an innovative method to automatically analyze artistic portraiture, capturing a high level of detail and accuracy of the portraits as well as the artists' individual style. Focused on devising a computational method to detect facial features in artwork, or facial landmarks such as eye corners or mouth corners, the researchers have successfully extended work done in photographs of natural face images to the artistic portraiture field.

"Because there is no training data of portrait paintings that contain facial landmarks, our key idea was to create such data using what we call 'artistic augmentation,'" says Professor Ariel Shamir, lead author of the study and dean of the Efi Arazi School of Computer Science at the Interdisciplinary Center Herzliya. "We transform photographic face data to be more similar to artistic portraits and use it to train new models for neural networks that work better for artistic inputs. By using our models, we not only gained better precision on portraits, but we also did not lose much accuracy on natural faces."

Shamir, along with his students Jordan Yaniv and Yael Newman, are set to present their work at SIGGRAPH 2019, held 28 July-1 August in Los Angeles. This annual gathering showcases the world's leading professionals, academics, and creative minds at the forefront of computer graphics and interactive techniques.

Landmark Points Guide the Style

The key differences between artistic portraits and natural facial images are twofold: texture appearance differences and geometric differences. Past work in this research area primarily focused on capturing the appearance style of art, and not the geometric style. With artistic portraits, geometric style is imperative, note the authors of the study, but challenging to capture since every artist has their own creative, distinct style.

"For example, the Italian painter Clemente Modigliani is known for painting elongated faces, and the American painter Margaret Keane is known for painting faces with very big, exaggerated eyes," says Shamir. "Our work allows computers and algorithms to reveal this information and recognize this aspect of geometric style in portrait paintings."

To capture geometric styles in portraits, there is a need to recognize the facial features and the structure of the face in the painting. To this end, the researchers' method concentrates on detecting facial features in the artwork, using landmark points on the face. Depending on the artist's style, these features can be diverse in shape and exaggerated, not resembling real human faces. To overcome this challenge, they apply a known method called "augmentation" of natural face images, transforming photographs of natural facial images to be more similar to "artistic" portraits, and train neural networks to detect the landmark points.

The researchers evaluated their landmark detection method by creating a dataset of artistic faces containing 160 artistic portraits by 16 different artists of various genres and styles, with large variations in both geometry and texture. In the paper outlining their work, they also demonstrate several applications for artistic facial feature detection and geometric style analysis. These include understanding the style of specific artists, comparing styles of different artists, and following possible trends of artistic styles. Another popular application is style-transfer: where one can transform a given input picture of a face to a painting in the style of a given artist in both texture and geometry.

In future work, the team hopes to use the geometric style signature to build classifiers that can recognize a specific artist and expand the definition of geometric style beyond faces—a current challenge in computer graphics and art.