This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:



trusted source


AI tool generates video from brain activity

AI tool generates video from brain activity
Brain decoding & video reconstruction. We propose a progressive learning approach to recover continuous visual experience from fMRI. High-quality videos with accurate semantics and motions are reconstructed. Credit: arXiv (2023). DOI: 10.48550/arxiv.2305.11675

"Alexa, play back that dream I had about Kirsten last week." That's a command that may not be too far off in the future, as researchers close in on technology that can tap into our minds and retrieve the imagery of our thoughts.

Researchers at the National University of Singapore and the Chinese University of Hong Kong reported last week that they have developed a process capable of generating video from . The research is published on the arXiv preprint server.

Using a process called imaging (fMRI), researchers Jiaxin Qing, Zijiao Chen and Juan Helen Zhou coupled data retrieved through imaging with the deep learning model Stable Diffusion to create smooth, high quality videos.

Successful recreations of still imagery gleaned from through AI-assisted Stable Diffusion, commonly used for image generation, have been reported in recent months. But, as Qing and his team reported, retrieving continuous visual images poses a particular challenge.

The process of fMRI generally taps blood oxygenation level-dependent (BOLD) signals and captures images of brain activity once every few seconds. This would yield poor-quality video images. A standard video capture rate is 30 frames per second.

"Understanding the information hidden within our complex brain activities is a big puzzle in ," Qing said. "The task of recreating from brain recordings, especially using non-invasive tools like fMRI, is an exciting but difficult task."

His team achieved high-caliber video with their Mind-Video model. Described as "a two-module pipeline designed to bridge the gap between image and video brain decoding," its fMRI decoder progressively learns from acquired brain signals, training with image databases and fine tuning.

The results were high-quality videos, they said, complete with motion and scene dynamics at an accuracy rate of 85%.

Qing says their work shows promise for future large model applications "from neuroscience to brain-computer interfaces."

The use of AI along with MRI and EGM (electromyogram) to study imagery, brain activity and muscle movement is opening new vistas into the workings of the mind. Shinji Nishimoto, a neuroscientist at Osaka University, says these processes may one day be used to capture thoughts and dreams.

Dream researcher Daniel Oldis, working with a colleague at the Cognitive Neuroscience Lab at the University of Texas, Austin, is working on dream-recording MRI technology that tracks activity and nerve impulses in muscles to define images, speech and movement in dreams.

"This is like the early years of the space race," he said recently. "But in this case, we're going into the dream space."

The has up to six dreams a night, but forgets 90% of them within minutes after awakening. The prospect of capturing a permanent record of dreams is exhilarating.

Dreams have mystified us for centuries. In early cultures, dreams were believed to be messages from God. In the 20th century, Freud introduced the idea of repressed sexual and emotional impulses behind our dreams.

Some of our greatest inspirations come from those nocturnal journeys we all take each night of our lives.

The melody to the Beatles' "Yesterday," named the best song of the 20th century by a BBC Radio poll, came to Paul McCartney in a dream. The famous three-note opening riff on the Rolling Stones' giant hit "Satisfaction" came to Keith Richards in the middle of the night; he awoke to record a rough version of the song, sleepily mumbled "I can't get no satisfaction," went back to sleep, and only realized what he had jotted down when he listened to the tape that morning.

Salvator Dali referred to fantastical surrealist works such as his "The Persistence of Memory" as "hand-painted dream photographs."

And a nervous student fearful that his admission to Stanford University was an error had an anxiety dream in which he downloaded the entire web onto his computer. When he awakened, he embarked on a project that one day would draw 89 billion viewers per month and answer 99,000 inquiries per second. His name: Larry Page. His creation: Google.

Recording and streaming our dreams—from nightmares to game-changing inspirational ideas—may be only a right now.

But as the works by Qing, Oldis and others indicate, not for long.

More information: Zijiao Chen et al, Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity, arXiv (2023). DOI: 10.48550/arxiv.2305.11675

Journal information: arXiv

© 2023 Science X Network

Citation: AI tool generates video from brain activity (2023, May 25) retrieved 3 March 2024 from
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Explore further

How has the COVID-19 pandemic affected our dreams?


Feedback to editors