Credit: Zeng, van der Lubbe & Loog.

Researchers at TU Delft in the Netherlands have recently developed a convolutional neural network (CNN)-based model to reconstruct drawings that have deteriorated over time. In their study, published in Springer's Machine Vision and Applications, they specifically used the model to reconstruct some of Vincent Van Gogh's drawings that were ruined over the years due to ink fading and discoloration.

"The Netherlands has an international reputation with respect to arts, with famous artists like Rembrandt, Mondrian and Van Gogh," Jan van der Lubbe, one of the researchers who carried out the study, told TechXplore. "Therefore, art and research into how to preserve cultural heritage play an important role in the Netherlands."

In recent years, a growing number of researchers have tried to develop machine learning techniques, such as CNNs, for the analysis of artworks. So far, these tools have primarily been used to identify the artist who created specific artworks or to determine whether paintings are real or fake.

In contrast with previous research, van der Lubbe and his colleagues set out to investigate the use of machine-learning techniques for the pixel-wise reconstruction of deteriorated paintings. When it comes to art preservation, the deterioration of paintings and drawings is a key challenge, so tools that can automatically reconstruct incomplete or ruined artworks would greatly simplify the work of art historians.

The team of researchers at TU Delft trained their CNN-based model on reproductions of deteriorated drawings by post-impressionist painter Van Gogh. In fact, some of Van Gogh's ink drawings have deteriorated significantly over the past century, and art historians have often tried to reproduce them.

These drawings cannot currently be exhibited, and in a few decades they may deteriorate completely. With this in mind, Van der Lubbe and his colleagues wanted to develop a model that can automatically reconstruct these invaluable artworks in order to preserve them and make them accessible to the public.

Credit: Zeng, van der Lubbe & Loog.

"One of the main goals of our research was to predict the original, past and future appearances of artworks on paper by means of machine-learning methods that integrate the results of both in-depth studies of the colors used and their discoloration over time," van der Lubbe said. "This could help to envisage how, for example, a by Van Gogh may have looked at the time of its creation."

The approach devised by van der Lubbe and his colleagues combines techniques for multi-resolution image analysis and deep CNNs to predict the past appearances of drawings pixel-wise. CNNs are algorithms inspired by biological neural networks such as those in the human brain that can be trained to complete specific tasks by analyzing large amounts of data.

"To our best knowledge, there are no or very few prior studies on the usage of machine-learning methods for the digital reconstruction of artworks," van der Lubbe said. "That is the key idea driving our research and the use of machine learning to reconstruct artworks. From earlier studies where we have considered different machine learning algorithms, convolutional neural network (CNN) approaches seemed most promising."

In their study, the researchers specifically trained a CNN to digitally reconstruct faded Van Gogh drawings on paper. The algorithm was trained on a dataset containing reproductions of the original drawings of varying quality, made at different times during past century.

"The examples we used in our study are reproductions of Van Gogh drawings in which the content and color have faded less severely, thus they are closer to the original drawing made by Van Gogh," van der Lubbe said. "We obtained the original drawings and reproductions from the Van Gogh Museum collection."

In addition to revealing how drawings looked in the past, the approach proposed by van der Lubbe and his colleagues could help art historians to identify appropriate conservation and restoration strategies, as well as effective practices for the preservation and display of artworks.

Credit: Zeng, van der Lubbe & Loog.

The researchers evaluated their model in a series of experiments and found that it achieved remarkable results. Their findings highlight the feasibility of using machine learning for the predictive reconstruction of degraded images, documents and artworks. Although the researchers specifically used their model to reconstruct Van Gogh's drawings, it could also be applied to other deteriorating artworks on paper or to 19th century manuscripts.

"We obtained better results for the digital reconstruction of Van Gogh drawings than those achieved until now using other methods," van der Lubbe said. "Of course, Van Gogh was only a test or example. Our technique could also extend beyond Van Gogh's drawings to drawings by other artists, paintings and old documents."

In the future, the tool developed by van der Lubbe and his colleagues could help art historians to create realistic reconstructions of artworks that might otherwise deteriorate completely. In their recent study, the researchers focused on one drawing at a time, training their CNN on a limited number of reproductions. However, the model could also be used to predict what the original drawing looked like based on a far larger amount of reproductions.

In addition, this technique currently works by analyzing visual information. In their next studies, the researchers would like to investigate whether analyzing both visual and chemistry-related information (e.g., the composition of the ink and its degradation rate) can enhance the model's performance.

"In the present study, we had a degraded present-day drawing," van der Lubbe said. "We feel that it would also be a great challenge to reconstruct the original drawing, particularly in cases where the original is not available or has disappeared, thus we only have reproductions from the past."

More information: Y. Zeng et al. Multi-scale convolutional neural network for pixel-wise reconstruction of Van Gogh's drawings, Machine Vision and Applications (2019). DOI: 10.1007/s00138-019-01047-3