Food-tracking AI system developed to reduce malnutrition in LTC homes

New technology could help reduce malnutrition and improve overall health in long-term care homes by automatically recording and tracking how much food residents consume.

The smart system, developed by researchers at the University of Waterloo, the Schlegel-UW Research Institute for Aging and the University Health Network, uses artificial intelligence software to analyze photos of plates of food after residents have eaten.

The sophisticated software, which examines color, depth, and other photo features, can estimate how much of each kind of food has been consumed and calculate its nutritional value.

"Right now, there is no way to tell whether a resident ate only their protein or only their carbohydrates," said Kaylen Pfisterer, who co-led the research while earning a Ph.D. in systems design engineering at Waterloo.

"Our system is linked to recipes at the long-term care home and, using artificial intelligence, keeps track of how much of each food was eaten to make sure residents are meeting their specific nutrient requirements."

It is estimated that more than half of residents of long-term care homes are either malnourished or at risk of malnutrition.

Food intake is now primarily monitored by staff who manually record estimates of consumption by looking at plates once residents have finished eating.

Robert Amelard, a Waterloo alumnus and postdoctoral fellow at University Health Network, said the subjectivity of that process results in an error rate of 50 percent or more. By comparison, the automated system is accurate to within five percent, "providing fine-grained information on consumption patterns."

Researchers collaborated with personal support workers, dietitians and other long-term care workers to develop the system, which saves time as well as improves accuracy and would ideally be added to tablet computers already used by front-line staff to keep electronic records.

"My vision would be to monitor and leverage any changes in food intake trends as yellow or red flags for the health status of residents more generally and for monitoring infection control," said Pfisterer, now a scientific associate at the University Health Network Centre for Global eHealth Innovation.

The research team also included Heather Keller, a professor of kinesiology and health sciences, Alexander Wong, a systems design engineering professor, and students Audrey Chung, Braeden Syrnyk and Alexander MacLean.

A paper on their work, "Automated food intake tracking requires depth-refined semantic segmentation to rectify visual-volume discordance in long-term care homes," appears in the journal Scientific Reports.