Cow goes moo: Artificial intelligence-based system associates images with sounds

Cow goes moo: Artificial intelligence-based system associates images with sounds
Credit: Disney Research

The cow goes "moo." The pig goes "oink." A child can learn from a picture book to associate images with sounds, but building a computer vision system that can train itself isn't as simple. Using artificial intelligence techniques, however, researchers at Disney Research and ETH Zurich have designed a system that can automatically learn the association between images and the sounds they could plausibly make.

Given a picture of a car, for instance, their system can automatically return the sound of a car engine.

A system that knows the sound of a car, a splintering dish, or a slamming door might be used in a number of applications, such as adding sound effects to films, or giving audio feedback to people with visual disabilities, noted Jean-Charles Bazin, associate research scientist at Disney Research.

To solve this challenging task, the research team leveraged data from collections of videos.

"Videos with audio tracks provide us with a natural way to learn correlations between sounds and images," Bazin said. "Video cameras equipped with microphones capture synchronized audio and visual information. In principle, every video frame is a possible training example."

One of the key challenges is that videos often contain many sounds that have nothing to do with the visual content. These uncorrelated sounds can include background music, voice-over narration and off-screen noises and sound effects and can confound the learning scheme.

"Sounds associated with a video image can be highly ambiguous," explained Markus Gross, vice president for Disney Research. "By figuring out a way to filter out these extraneous sounds, our research team has taken a big step toward an array of new applications for ."

"If we have a video collection of cars, the videos that contain actual sounds will have audio features that recur across multiple videos" Bazin said. "On the other hand, the uncorrelated sounds that some videos might contain generally won't share any redundant features with other videos, and thus can be filtered out."

Once the video frames with uncorrelated sounds are filtered out, a computer algorithm can learn which sounds are associated with an image. Subsequent testing showed that when presented an image, the proposed system often was able to suggest a suitable sound. A user study found that the system consistently returned better results than one trained with the unfiltered original collection.

Combining creativity and innovation, this research continues Disney's rich legacy of inventing new ways to tell great stories and leveraging technology required to build the future of entertainment.

These results were recently presented at a European Conference on Computer Vision (ECCV) workshop in Amsterdam. In addition to Jean-Charles Bazin, the research team included Matthias Solèr and Andreas Krause of ETH Zurich's Computer Science Department, and Oliver Wang and Alexander Sorkine-Hornung of Disney Research.


Explore further

Artificial intelligence produces realistic sounds that fool humans

More information: "Suggesting Sounds for Images from Video Collections-Paper" [PDF, 4.93 MB]
Provided by Disney Research
Citation: Cow goes moo: Artificial intelligence-based system associates images with sounds (2016, November 15) retrieved 18 August 2019 from https://techxplore.com/news/2016-11-cow-moo-artificial-intelligence-based-associates.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.
12 shares

Feedback to editors

User comments

Nov 15, 2016
How come AI Cannot say WHY Carnivores don't commit Suicide like some Humans Do....I mean when They die of Hunger without hurting meek animals, Cannot Eat Grass, haven't heard of Agriculture and What Not!

Nov 15, 2016
How come AI Cannot say WHY Carnivores don't commit Suicide like some Humans Do....I mean when They die of Hunger without hurting meek animals, Cannot Eat Grass, haven't heard of Agriculture and What Not!

It is hightime To Trash the Word, Almighty B.S; Hurricanes, Floods, Tornadoes, Tsunamis, Epidemics, So What? B.S UnScientific Term Indeed!

Nov 15, 2016
How come AI Cannot say WHY Carnivores don't commit Suicide like some Humans Do....I mean when They die of Hunger without hurting meek animals, Cannot Eat Grass, haven't heard of Agriculture and What Not!

Do you still think that they will understand the hazard of hurting, if you Catch one during its act of Predator-ship & TEAR of its Skin with Scissors?

Nov 15, 2016
Can AI Order Distilled Water from Sewage? It has to Entangle with H2o Molecules disregarding EVERYTHING ELSE Without using Heat!

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more