Machine learning enhances non-verbal communication in online classrooms

Machine learning enhances non-verbal communication in online classrooms
If concertmaster (top left) looks at her camera, it will appear to the musicians that she is looking at all of them. But gaze tracking system shows she is looking at Walter (second row center), and labels the concertmaster’s video feed with the cue “Walter” so the entire class knows intended gaze recipient. The cue updates continuously when she looks at another musician – establishing non-verbal communication with the ensemble. Credit: University of California - San Diego

Researchers in the Center for Research on Entertainment and Learning (CREL) at the University of California San Diego have developed a system to analyze and track eye movements to enhance teaching in tomorrow's virtual classrooms—and perhaps future virtual concert halls.

UC San Diego music and computer science professor Shlomo Dubnov, an expert in computer music who directs the Qualcomm Institute-based CREL, began developing the new tool to deal with a downside of teaching music over Zoom during the COVID-19 pandemic.

"In a music classroom, non-verbal communication such as facial affect and body gestures is critical to keep students on task, coordinate musical flow and communicate improvisational ideas," said Dubnov. "Unfortunately, this non-verbal aspect of teaching and learning is dramatically hampered in the virtual classroom where you don't inhabit the same physical space."

To overcome the problem, Dubnov and Ph.D. student Ross Greer recently published a conference paper on a system that uses eye tracking and machine learning to allow an educator to make 'eye contact' with individual students or performers in disparate locations—and lets each student know when he or she is the focus of the teacher's attention.

The researchers built a and undertook a in a virtual music class at UC San Diego via Zoom.

"Our system uses a camera to capture the presenter's eye movements to track where they are looking on screen," explained Greer, an electrical and computer engineering Ph.D. student in UC San Diego's Jacobs School of Engineering. "We divided the screen into 91 squares, and after determing the location of the teacher's face and eyes, we came up with a 'gaze-estimation' algorithmm that provides the best estimate of which box—and therefore which student—the teacher is looking at."

As the system recognizes a change in where the teacher is looking, the algorithm determines the identity of the and tags his or her name on screen so everyone knows whom the presenter is focusing on.

In the pilot study, Dubnov and Greer found the system to be highly accurate in estimating the presenter's gaze—managing to get within three-quarters of an inch (2cm) of the correct point on a 27.5 x 13 inches (70x39cm) screen. "In principle," Greer told New Scientist magazine, "the system should work well on small screens, given enough quality data."

One downside, according to Dubnov: the further the presenter is from the camera, the eyes become smaller and harder to track, resulting in less accurate gaze estimation. Yet with better training data, higher-quality camera resolution, and further advances in tracking facial and body gestures, he thinks the system could even allow the conductor to wield a baton remotely and conduct a distributed symphony orchestra—even if every musician is located somewhere else.


Explore further

Q&A: Artificial intelligence and the classroom of the future

More information: Ross Greer et al, Restoring Eye Contact to the Virtual Classroom with Machine Learning, Proceedings of the 13th International Conference on Computer Supported Education (2021). DOI: 10.5220/0010539806980708
Citation: Machine learning enhances non-verbal communication in online classrooms (2021, June 22) retrieved 21 September 2021 from https://techxplore.com/news/2021-06-machine-non-verbal-online-classrooms.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.
29 shares

Feedback to editors