Integration propels machine vision: Research reviews in-sensor visual perception and inference

A joint research team in China wrote a review on in-sensor visual computing, a three-in-one hardware solution that is more efficient, economical and secure than conventional machine vision systems, which collect, store, and interpret visual signals on separate hardware units. This review was published in Intelligent Computing.

In-sensor visual computing systems are inspired by how humans and other mammals collect, extract and process visual signals, an intricate biological mechanism showing low latency and low energy cost. By integrating sensing, storage and computation onto the focal plane of image sensors, in-sensor visual computing systems process data within each sensor and extract only critical information from raw signals, rather than processing whole image data like conventional systems.

Therefore, they have the potential to overcome the three major obstacles—high latency, high power consumption and privacy risks—that hinder the further development of their conventional counterparts.

The development of in-sensor computing devices has concentrated on novel circuit designs and new materials. The review centers on a vision chip with a novel circuit design called the SCAMP pixel processor array or SCAMP chip, which is relatively mature among emerging sensors and "an interdisciplinary and fertile research platform" for relevant research. First developed two decades ago, ever-improving focal-plane sensor-processors like the SCAMP chip have been widely used in computing experiments, but not thoroughly surveyed.

The authors first introduce the most up-to-date system based on the SCAMP chip, SCAMP-5d. It is a general-purpose, programmable, immensely parallel system extensively used in robotics and computer vision. Software tools and platforms developed for the SCAMP chip are also introduced, including development frameworks for programming the chip, semi-simulated and fully-simulated platforms for simulating the chip's operations and kernel filter compilers for optimizing visual processing algorithms.

Next, the authors give an overview of in-sensor visual computing algorithms and applications based on the versatile SCAMP chip. They survey algorithms ranging from lower-level image processing techniques, such as image enhancement and feature extraction, to higher-level tasks such as classification, localization and segmentation using neural networks. The applications enabled by these algorithms are mainly state estimation and robot navigation.

Although the in-sensor visual computing systems using the SCAMP pixel processor array have brought about extensive technological advances, they still have limitations such as low resolution, scarce computing resources, noise and unsatisfactory algorithm design and deployment. Compensating for current limitations while exploring other non-conventional computing methods such as sensor fusion and edge computing, engineers and researchers of next-generation SCAMP vision systems are trying to turn such obstacles into opportunities.

The authors themselves are actively involved in the "co-development and co-optimization of circuit design, integration technologies and associated algorithms" for academic and commercial purposes. They believe that next-generation SCAMP vision systems will show better performance at lower power consumption.

The authors are Yanan Liu of Shanghai University, Rui Fan of Tongji University, Jianglong Guo of Harbin Institute of Technology, Hepeng Ni of Shandong Jianzhu University, and M. Usman Maqboo Bhutta of The Chinese University of Hong Kong.