Grand Theft Auto and AI help team turn dog pics into 3D models

Photographs of dogs could soon be used to help generate 3D models more accurately than ever before—thanks to an award-winning study from the University of Surrey and the famous video game Grand Theft Auto.

The researchers taught an artificial intelligence (AI) system to predict the 3D pose from a 2D image of a dog—which they trained on images they created using Grand Theft Auto V.

"Our model was trained on CGI dogs, but we were able to use it to make 3D skeletal models from photographs of real animals. That could let conservationists spot injured wildlife, or help artists create more realistic animals in the metaverse," said Moira Shooter, Postgraduate Research Student.

One way to teach AI to get 3D information from 2D images is to show it photos while giving it information about 3D "ground truth"—where the objects actually are in 3D space. For humans, that means wearing motion capture suits.

Even on their best behavior, getting dozens of dogs to wear motion capture suits can prove challenging.

Instead, researchers created a myriad of virtual dogs to study.

Researchers altered the code of Grand Theft Auto, switching the main character for one of eight kinds of dog—a process known as "modding." They generated 118 videos of the dogs sitting, walking, barking and running in a range of different weather and lighting conditions.

The team called their new database DigiDogs—made up of 27,900 frames. They will now fine-tune it using Meta's DINOv2 model to make sure it can predict a 3D pose just as well as real dog pictures.

"3D poses contain so much more information than 2D photographs. From ecology to animation—this neat solution has so many possible uses," said Shooter.

The research won the prize for Best Paper at the IEEE/CVF's Winter Conference on Applications of Computer Vision.