New mathematical models and algorithms for intelligent image processing inspired by the human visual cortex

New mathematical models and algorithms for intelligent image processing inspired by the human visual cortex
How the algorithm changes the images to be "more beautiful." Credit: Pock - TU Graz/ICG

Using the visual cortex as a model in the human brain, the research group led by ERC-award-winner Thomas Pock has developed new mathematical models and algorithms as the basis for faster and more intelligent image processing programs.

Our can capture images and recognize objects in a fraction of a second, even if they are barely visible or only fragmentary. One reason for this fantastic peak performance is the highly efficient hierarchical layer architecture of the visual cortex. It filters the visual information, recognizes connections and completes the image using familiar patterns. The process behind this is still hardly understood in its complexity. It is true that now exist that can match or, in some cases, exceed on certain pattern recognition tasks. One disadvantage of these algorithms, however, is that it is hard to understand what they have learned, how they work, or when they make mistakes.

Thomas Pock from the Institute of Computer Graphics and Vision at Graz University of Technology (TU Graz) was on the trail of this knowledge as part of his ERC Starting Grant project HOMOVIS (High Level Prior Models for Computer Vision). He worked intensively on the question of how known modes of operation of the visual cortex can be calculated using mathematical models and transferred to image processing applications. Five years of research, 41 publications and one patent later, the researcher and his research group have accumulated extensive knowledge that enables new image processing algorithms for a wide variety of applications.

Suggestions from Wertheimer and Euler

Pock based his work on Max Wertheimer's Gestalt laws of perception. The main founder of Gestalt psychology used these laws to try to explain the process of human vision, in which stimuli and sensory impressions are put together to form a large whole. "Humans can already correctly recognize partial or incomplete objects on the basis of single points or subjective contours (illusory contours). The automatically fills in the missing image information. For example, by connecting the points via curves that are as smooth as possible," says Pock. Pock and his team described this phenomenon of shape finding for the first time using mathematical models based on Euler's elastic curves—a famous equation by the mathematician Leonhard Euler that can be used to calculate curves of minimum curvature.

New mathematical models and algorithms for intelligent image processing inspired by the human visual cortex
The portrait of Albert Einstein was reconstructed from incomplete data with the help of the Elastica model. Credit: Ferdinand Schmutzer, 1921. Creative Commons Public Domain

Representation in a higher dimensional space

Based on Euler's elastic curves, Pock's group developed new algorithms to solve certain curvature-dependent image processing problems. Consequently, the solution is all the easier if the (2D) images and their features are represented as data points in three-dimensional space. "In the third dimension, we get an additional variable with the orientation of the object edges," Pock explains. This, too, is modeled on human vision and goes back to the pioneering work of two Nobel laureates, David Hubel and Torsten Wiesel, who established in 1959 that the visual cortex is composed of orientation-sensitive layers.

From a mathematical and computer science point of view, the biggest advantage of this three-dimensional embedding is that image processing problems can be solved using convex optimization algorithms. In mathematical optimization, the boundary between convex and non-convex optimization is considered as the great barrier that distinguishes solvable from unsolvable problems. "Thus, we are guaranteed to be able to calculate the best image for all the given input images—of course, only with respect to the used," says Pock.

Future Outlook

Now, Pock and his team are working on improved models that combine the known structural properties of the visual cortex with deep-learning algorithms. The goal is to develop models that perform as well as current deep-learning algorithms, but also allow a deeper understanding about the structures learned. Initial successes have already been achieved in the reconstruction of computer tomography and magnetic resonance images. "With the newly developed algorithms, it is now possible to reconstruct images with the highest quality despite less data being recorded. This saves time and computing power, and thus also costs," explains Pock.

Citation: New mathematical models and algorithms for intelligent image processing inspired by the human visual cortex (2021, June 1) retrieved 4 March 2024 from
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Explore further

Cardiovascular diseases: New computer model improves therapy


Feedback to editors