Projection system shines makeup on actors during live performances

The facial appearance of actors can be transformed during live stage performances using a new advanced system developed by a team at Disney Research that can track an actor's movements and changing expressions so that the face can be painted with light, rather than physical makeup.

The projection system can track the actor's facial movements without use of facial markers. By adjusting illumination, the system can display any color or texture that an artist wants to achieve. That could include effects that make the actor appear older, or appear to grow facial hair, or have clownish face paint—all in real-time during live performances.

"We've seen astounding advances in recent years in capturing facial performances of actors and transferring those expressions to virtual characters," said Markus Gross, vice president at Disney Research. "Leveraging these technologies to augment the appearance of live actors is the next step and could result in amazing transformations before our eyes of stage actors in theaters or other venues."

In contrast to previous work, the Disney team showed it can track facial movements and dynamically adjust the projections to a wide variety of arbitrary movements, rather than project augmentations onto static objects or rigidly constrain facial movements, said Anselm Grundhöfer, principal research engineer at Disney Research

The researchers will present their software and hardware system, called Makeup Lamps, April 24 at the European Association for Computer Graphics conference, Eurographics 2017, in Lyon, France.

"The key challenge of live augmentation is latency—the time between generating an image that matches the actor's pose and when the image is displayed," Grundhöfer said. "The larger the latency, the more the actor's pose will have changed and the greater the potential misalignment between the augmentation and the face."

In designing Makeup Lamps, the research team aimed to reduce latency at each step of the process—from capturing the facial pose, to processing to projection, he said. To do so, the team limited the complexity of its algorithms. They also employed a coaxial camera-projection setup, where the camera that detects facial movements shares the same optical axis as the projector that illuminates the face. This enabled the team to process images in two dimensions, rather than three, and still provide consistent augmentation of the face.

Some latency is unavoidable, of course, so the team used a method called Kalman filtering that uses measurements over time to make predictions, enabling slight adjustments that better align the augmentations with the facial performance.

Makeup Lamps is able to simulate different lighting conditions and facial effects, such as wrinkles, that are expression specific while allowing the performance to change over time," Grundhöfer said.

"We believe that projection-based non-rigid expressive augmentation could give rise to a wide variety of creative applications in the near future," he added. For instance, such as system might be used to simulate cosmetic plans on a face before actually applying makeup.