Team realizes brain's sensory functions using artificial synapse devices

Researchers from the College of Electronic Information and Optical Engineering at Nankai University have used flexible artificial synapse devices to develop a neuromorphic motion perception system that realizes the brain's multisensory functions at the hardware level and exhibits excellent motion perception performance.

Dr. Jiang Chengpeng, from the College of Electronic Information and Optical Engineering at Nankai University, is the first author, and Professor Xu Wentao, from the College of Electronic Information and Optical Engineering at Nankai University, is the corresponding author of the paper "Mammalian-brain-inspired neuromorphic motion-cognition nerve achieves cross-modal perceptual enhancement," which has been published in Nature Communications.

The design of the neuromorphic motion perception system is inspired by the macaque's multisensory integration and spatial perception mechanisms. A macaque's self-motion will stimulate motion information such as inertial signals and optical flow signals in the vestibule and the retina. The specific areas of the cerebral cortex will process and identify the motion information encoded as spike pulse, and then realize spatial perception by integrating information from different sensory modalities.

Accelerometers and gyroscopes in the neuromorphic motion perception system acquire acceleration and angular velocity signals, respectively, which are encoded into two spike trains that are transmitted to high-performance synaptic transistors for processing. The correlation between the two pulse sequences as well as their temporal relationship affect the synaptic plasticity of the device, thereby affecting the device output. The motion signals are classified and identified by evaluating the average firing rate of the pulse and the output current of the synaptic device.

Furthermore, an optical flow sensor, a vibrotactile sensor, and an inertial sensor comprise a sensing unit that can detect sensory information of visual, tactile and acceleration modes. Information from different types of sensors can be effectively integrated, thereby significantly enhancing the accuracy of motion recognition (higher than 94%). The experimental results are consistent with the perceptual enhancement effect of the brain. Moreover, the system can be attached to human skin or equipped on small drones to perform complex tasks such as recognition of human motion and drone flight pattern thanks to its features of wearability, high integration and low power consumption.

Essentially, the system mimics the process of sensory cue integration in the mammalian brain, and realizes brain-like motion perception by employing the spike coding strategy for sensory signals, the spike-integration characteristics of synaptic devices, and the spatiotemporal recognition method of synaptic current signals. This work combines neuromorphic cognitive intelligence with the brain's multimodal perception mechanism, and is of guiding significance for the development of brain-inspired devices and biomimetic electronics. It can be applied to various fields such as mobile robots, intelligent wearable devices, and human-machine interface.