The researchers addressed three problems: (1) Acting like a dog: where the goal is to predict the future movements of the dog given a sequence of previously seen images. (2) Planning like a dog: where the goal is to find a sequence of actions that move the dog between the locations of the given pair of images. (3) Learning from a dog: where we use the learned representation for a third task (e.g., walkable surface estimation). Credit: arXiv:1803.10827 [cs.CV]

A team of researchers from the University of Washington and the Allen Institute for AI has trained an AI system to respond like a dog using data from an actual animal. In their paper uploaded to the arXiv preprint server, the group describes their system and what it can and cannot do. The team is also going to present their work at the Conference on Computer Vision and Pattern Recognition this summer.

AI systems are typically based on deep-learning algorithms that process data describing events, and then using what they have learned to predict future behavior. In this new effort, the researchers have applied such a strategy to dog behavior. Their goal was to build an AI system that could respond in ways similar to a dog under given circumstances. To achieve this goal, they affixed a host of sensors to a malamute named Kelp M. Redmon. They put a GoPro and microphone on his head, inertia sensors on his body, legs and tail, and an Arduino unit on his back to collect and process the data as it streamed in. Then they let the dog do dog stuff, like play at the park.

The AI system was set up to achieve three main goals: predict future movements, plan a task and learn from dog behavior. The idea was to have the system learn to predict what a dog would do next in a given scenario, such as when spotting a squirrel. To mimic a dog, the system would need to create a plan of action to carry out the predicted future movements and would, of course, have to be able to learn how to do dog things by learning how an actual dog does them.

In all, the team collected 24,500 frames of video, which were synchronized with body moments and sound. They used 21,000 of those frames to train their AI system and the rest to test it. They report that the system works well, outperforming baselines on tasks they deemed challenging. The AI system was not connected to a dog robot, but that is clearly the direction the research is heading—it likely will not be long before AI behavioral systems are linked with projects that have focused on getting dog-like robots to move like real animals in useful ways.

More information: Who Let The Dogs Out? Modeling Dog Behavior From Visual Data, arXiv:1803.10827 [cs.CV] arxiv.org/abs/1803.10827

Abstract
We introduce the task of directly modeling a visually intelligent agent. Computer vision typically focuses on solving various subtasks related to visual intelligence. We depart from this standard approach to computer vision; instead we directly model a visually intelligent agent. Our model takes visual information as input and directly predicts the actions of the agent. Toward this end we introduce DECADE, a large-scale dataset of ego-centric videos from a dog's perspective as well as her corresponding movements. Using this data we model how the dog acts and how the dog plans her movements. We show under a variety of metrics that given just visual input we can successfully model this intelligent agent in many situations. Moreover, the representation learned by our model encodes distinct information compared to representations trained on image classification, and our learned representation can generalize to other domains. In particular, we show strong results on the task of walkable surface estimation by using this dog modeling task as representation learning.

Journal information: arXiv