New digital-camera-based system can 'see' around corners

What if your car possessed technology that warned you not only about objects in clear view of your vehicle—the way that cameras, radar, and laser can do now in many standard and autonomous vehicles—but also warned you about objects hidden by obstructions. Maybe it's something blocked by a parked car, or just out of sight behind a building on a street corner.

This ability to see things outside your line of sight sounds like science fiction, but researchers have made strides in the last decade to bring what's called "non-line-of-sight imaging" to reality.

Until now, they've had to rely on expensive and stationary equipment. But Vivek Goyal and a team of researchers from Boston University have developed a system that, employing a computer algorithm and a simple digital camera, can give us a more affordable and agile look at what's around the corner.

"There's a bit of a research community around non-line-of-sight imaging," says Goyal, a Boston University associate professor of electrical and computer engineering. "In a dense urban area, if you could get greater visibility around the corner, that could be significant for safety. For example, you might be able to see that there's a child on the other side of that parked car. You can also imagine plenty of scenarios where seeing around obstructions would prove extremely useful, such as taking surveillance from the battlefield, and in search and rescue situations where you might not be able to enter an area because it's dangerous to do so."

In a paper published in Nature on January 23, 2019, Goyal and a team of researchers say they are able to compute and reconstruct a scene from around a corner by capturing information from a digital photograph of a penumbra, which is the partially shaded outer region of a shadow cast by an opaque object.

"Basically, our technique allows you to see what's around the corner by looking at a penumbra on a matte wall," Goyal says.

When shadows turn ordinary walls into mirrors

Against a matte wall, Goyal explains, light scatters equally rather than being concentrated or reflected back in one direction like a mirror. Normally, that wouldn't give enough organized information for a computer program to translate what's happening in a visible scene around the corner. But Goyal's team discovered that when there is a known solid object around the corner, the partially obstructed scene creates a blurry penumbra. The object can really be anything as long as it's not see-through. In this case, the researchers opted to use an ordinary chair. To the human eye, the resulting penumbra may not look like much. For a computer program, it's highly informative.

By inputting the dimensions and placement of the object, the team found that their computer program can organize the light scatter and determine what the original scene looks like—all from a digital photograph of a seemingly blurry shadow on a wall.

"Based on light ray optics, we can compute and understand which subsets of the scene's appearance influence the camera pixels," Goyal says, and "it becomes possible to compute an image of the hidden scene."

For their research purposes, they created different scenes by displaying different images on an LCD monitor. But Goyal explains there's nothing fundamental about using an LCD screen or not.

Could the image of a human being standing around the corner, for example, be reconstructed using their approach? Goyal says there's no conceptual barrier preventing it, but that they haven't tried it yet. They did, however, make additional scenes by cutting out colored pieces of construction paper and pasting them on foam board to see if their system could detect the shapes and colors. Goyal says their "kindergarten art project" scenes were indeed able to be interpreted.

Seeing potential all around

The most fundamental limitation is the contrast between the penumbra and the surrounding environment, Goyal explains. "The results we present are for a relatively darkened room," he says. When the team increased the levels of ambient light in the lab, they observed that the penumbra became harder to see and the system's ability to precisely reconstruct the around-the-corner scene gradually became worse.

Goyal says that while real-world applications for using non-line-of-sight imaging are still a ways off, the breakthrough is in the proof of concept.

"In the future, I imagine there might be some sort of hybrid method, in which the system is able to locate foreground opaque objects and factor that into the computational reconstruction of the scene," he says.

The most exciting aspect of their findings is the discovery that so much information can be extracted from penumbras, Goyal says, which are literally found everywhere.

"When you realize how much light can be extracted from them, you just can't look at shadows the same way again," he says.