Understanding how humans process light could improve autonomous driving at high speed, which requires improved machine vision to avoid potholes and other obstacles. Credit: Bryan-College Station Eagle/Laura McKenzie

A team of Army researchers uncovered how the human brain processes bright and contrasting light, which they say is a key to improving robotic sensing and enabling autonomous agents to team with humans.

To enable developments in autonomy, a top Army priority, machine sensing must be resilient across changing environments, researchers said.

"When we develop machine algorithms, real-world images are usually compressed to a narrower range, as a cellphone camera does, in a process called tone mapping," said Andre Harrison, a researcher at the U.S. Army Combat Capabilities Development Command's Army Research Laboratory. "This can contribute to the brittleness of machine vision algorithms because they are based on artificial images that don't quite match the patterns we see in the real world."

By developing a new system with 100,000-to-1 display capability, the team discovered the brain's computations, under more real-world conditions, so they could build biological resilience into sensors, Harrison said.

Current vision algorithms are based on human and animal studies with computer monitors, which have a limited range in luminance of about 100-to-1, the ratio between the brightest and darkest pixels. In the real world, that variation could be a ratio of 100,000-to-1, a condition called high dynamic range, or HDR.

A vehicle kicks-up mud as one of six vendors demonstrates the capabilities of a robotic combat vehicle at Texas A&M University System's RELLIS campus on Thursday, May 16, 2019. Credit: Bryan-College Station Eagle/Laura McKenzie

"Changes and significant variations in light can challenge Army systems—drones flying under a could be confused by reflectance changes when wind blows through the leaves, or autonomous vehicles driving on rough terrain might not recognize potholes or other obstacles because the lighting conditions are slightly different from those on which their vision algorithms were trained," said Army researcher Dr. Chou Po Hung.

The research team sought to understand how the brain automatically takes the 100,000-to-1 input from the real world and compresses it to a narrower range, which enables humans to interpret shape. The team studied early visual processing under HDR, examining how simple features like HDR luminance and edges interact, as a way to uncover the underlying brain mechanisms.

"The brain has more than 30 visual areas, and we still have only a rudimentary understanding of how these areas process the eye's image into an understanding of 3-D shape," Hung said. "Our results with HDR luminance studies, based on human behavior and scalp recordings, show just how little we truly know about how to bridge the gap from laboratory to real-world environments. But, these findings break us out of that box, showing that our previous assumptions from standard computer monitors have limited ability to generalize to the real world, and they reveal principles that can guide our modeling toward the correct mechanisms."

The Journal of Vision published the team's research findings, Abrupt darkening under (HDR) luminance invokes facilitation for high contrast targets and grouping by luminance similarity.

Researchers said the discovery of how light and contrast edges interact in the brain's visual representation will help improve the effectiveness of algorithms for reconstructing the true 3-D world under luminance, by correcting for ambiguities that are unavoidable when estimating 3-D shape from 2-D information.

Examples of high dynamic range luminance in views of a cave opening, where combinations of indoor and outdoor luminance can exceed a 10,000-to-1 maximum-to-minimum luminance ratio. The scene at the right is a blended image across multiple exposures, illustrating the human ability to see multiple targets (three uniforms and one car) across vast luminance differences in the same view. Credit: U.S. Army

"Through millions of years of evolution, our brains have evolved effective shortcuts for reconstructing 3-D from 2-D information," Hung said. "It's a decades-old problem that continues to challenge machine vision scientists, even with the recent advances in AI."

In addition to vision for autonomy, this discovery will also be helpful to develop other AI-enabled devices such as radar and remote speech understanding that depend on sensing across wide dynamic ranges.

With their results, the researchers are working with partners in academia to develop computational models, specifically with spiking neurons that may have advantages for both HDR computation and for more power-efficient vision processing—both important considerations for low-powered drones.

"The issue of dynamic range is not just a sensing problem," Hung said. "It may also be a more general problem in brain computation because individual neurons have tens of thousands of inputs. How do you build algorithms and architectures that can listen to the right inputs across different contexts? We hope that, by working on this problem at a sensory level, we can confirm that we are on the right track, so that we can have the right tools when we build more complex AIs."

More information: Chou P. Hung et al, Abrupt darkening under high dynamic range (HDR) luminance invokes facilitation for high-contrast targets and grouping by luminance similarity, Journal of Vision (2020). DOI: 10.1167/jov.20.7.9

Journal information: Journal of Vision