March 20, 2019 feature
A new approach to discover visual patterns in art collections
Researchers at UC Berkeley and Ecole des Ponts Paris Tech have recently developed a deep learning approach for discovering recurring visual patterns in art collections. Their paper, pre-published on arXiv, will be presented at CVPR 2019, a renowned computer vision event in June.
While every artwork might seem unique, artists often use recurrent visual elements or motifs (e.g. angels, windmills, etc.). For instance, critics believe that some paintings by Flemish painter Jan Brueghel were merely imitations or adaptations of his own works, as well as those of his father, Pieter Breughel.
In their research, art historians often try to map out visual connections between different artworks, as this could shed some light on their provenance and authorship. However, uncovering similar visual patterns in large art collections can be very challenging for humans and machines alike.
"We started this project following a presentation and discussion with an art historian, Elizabeth Honig, where she presented such correspondences and why they were important for her in her study of Brueghel works," Mathieu Aubry, one of the researchers who carried out the study, told TechXplore. "Our first goal was to make the job of art historians easier and more scalable by automatically identifying in digital image collections details that were directly copied between different works, despite small modifications and differences in the style of the representation (e.g., engraving, painting, drawing, etc.)."
In their recent study, Aubry and his colleagues proposed an approach that can automatically discover recurrent visual patterns in large art collections. Essentially, they trained an unsupervised machine learning model to find correspondences between near-duplicate visual elements across different artworks.
"The main novelty of our approach is to learn, without human supervision, a deep image descriptor specifically adapted to our task: matching exact copies across different representation styles," Aubry explained. "To do so, we introduce a procedure that validates candidate correspondences using spatial consistency between neighbour matches."
The researchers used the spatial consistency between neighbouring feature matches as a supervisory fine-tuning signal. This adapted feature leads to more accurate style-invariant matching. Combined with a standard discovery approach based on geometric verification, the feature allows their deep learning approach to identify duplicate patterns in large art datasets.
"Our CVPR work focused on the computer vision aspects. Collaborations with art historians to apply the method we developed to analyse artwork collections are still ongoing," Aubry said. "We think that it will really change both the scale and the type of study art historians will perform, by allowing them to look for and analyse connections between artworks at a much larger scale. Indeed, when trying to annotate connections for just a few details on a medium-scale dataset, we saw firsthand how tedious and costly such a process was to perform manually."
Aubry and his colleagues evaluated their method on several datasets, including the Oxford5K photo dataset and a newly annotated dataset of artworks attributed to the Brueghel family. In these evaluations, their approach attained remarkable results, outperforming other state-of-the-art techniques for uncovering visual patterns in artworks. In addition, their approach achieved state-of-the-art performance on the Large Time Gap Location dataset, effectively localizing historical architecture photographs and modern ones.
In the future, the deep learning approach devised by Aubry and his colleagues could assist art historians in discovering visual patterns across large art collections. According to the researchers, their approach can also be easily transferred to other problems, such as geo-localization and historical watermark recognition.
"We want to push the applications of our approach in humanities, by working directly with art historians to tune our method to their specific need and helping them to use it," Aubry said. "We also plan to work on extending idea of leveraging redundancy and spatial consistency in deep learning to different type of imagery and different type of applications."
© 2019 Science X Network