Scientists teach a neural network to recognize PC users' fatigue

A research team consisting of scientists from St Petersburg University, St Petersburg Federal Research Center of the Russian Academy of Sciences and some other organizations have created a database of eye movement strategies of operators monitoring objects on a PC screen in various states (tired/alert).

Based on the collected data, the scientists are planning to train neural network models that will form the basis of high-precision systems for functional state tracking to ensure safety on roads and industrial facilities.

Today, a large number of transport, industrial and defense facilities are controlled by operators, drivers or whole teams of professionals working at unified information centers. The ability to ensure the safety of these facilities often depends on the psychophysiological state of the staff. The professionals who can benefit from such a system include: drivers at vehicle fleets, aircraft pilots, air traffic controllers, industrial plant controllers, and so forth.

The scientists organized simultaneous registration of a set of behavioral and neurophysiological indicators. Their findings were published in the journal Sensors.

"An integrated approach provides a more complete picture and a more objective assessment of the functional state, in contrast to approaches involving separate registration of certain indicators that reflect the state of fatigue. Thus, a common method of cardiac time interval measurement used to register fatigue is quite controversial in terms of the accuracy of state assessment. It is based on the registration of heart rate indicators," said Irina Shoshina—Doctor of Biological Sciences, Professor of the Institute for Cognitive Research St. Petersburg State University.

"We used a unique approach based on comparing indicators of the nature of eye movements. The eye movements reflect the dynamics of the interaction of neural networks of static and dynamic vision with psychophysiological indicators of the functional state and psychological tests."

The scientists are planning to use the database to train a neural network that will be able to detect operator fatigue with high accuracy based on the eye movement strategies. According to Irina Shoshina, this approach will make it possible to remotely assess the severity of fatigue. The prepared database is in the public domain and is accessible to all software developers. They may use it to test their products.

"We have developed a comprehensive database suitable for training neural networks that classify a person's state as tired / alert. The collected database has a unique set of various labeled indicators. By using them, you can train neural networks to recognize the state of human fatigue with high accuracy," says Alexey Kashevnik, Project Manager, Senior Research Associate in the Laboratory of Integrated Automation Systems, St Petersburg Federal Research Center of the Russian Academy of Sciences

The information about the indicators reflecting the functional state was collected through a number of sensors such as: a video camera; an eye tracker; a heart rate monitor; and an electroencephalograph. In addition, as part of the experiment, the operators were tested for the quality of their sleep, fatigue, complex visual-motor reaction, etc.

Measurements were taken in the morning, afternoon and evening during the working day. The process was recorded on a video camera. The research lasted eight days and involved 10 people who were engaged in various activities, both passive (reading) and active (playing Tetris).