Stable Diffusion AI used to generate images from the brain

Share post:

A team of Japanese researchers led by Yu Takagi and Shinji Nishimoto from Osaka University’s Graduate School of Frontier Biosciences used Stable Diffusion AI to help generate images based on people’s brain activity using MRI scans as input.

The researchers relied on a latent diffusion model (LDM) known as Stable Diffusion to decode fMRI data from human brains and reconstruct visual experiences using a diffusion model (DM). According to the researchers, a few studies have produced high-resolution image reconstructions, but only after training and fine-tuning generative models. This resulted in limitations because complex model training is difficult and there aren’t many samples to work with in neuroscience. No other researchers had attempted to use diffusion models for visual reconstruction prior to this new study.

The researchers showed subjects a series of images and performed fMRI (functional magnetic resonance imaging) scans of their brains while they focused on the image. The final image is the result of several parts, including an existing fMRI image output and Semantic Decoder. The model was then processed, and noise was added to it via the diffusion process. Finally, the researchers used text representations decoded from fMRI signals in the higher visual cortex as input to create a final constructed image.

This simple framework uses Stable Diffusion to reconstruct high-resolution images from functional Magnetic Resonance Imaging (fMRI) signals, eliminating the need for training or fine-tuning of complex deep generative models. Stable Diffusion’s text-to-image conversion process incorporates the semantic information expressed by the conditional text while retaining the appearance of the original image.

The sources for this piece include an article in Vice.

Featured Tech Jobs



Related articles

Stack Overflow moderators strike over limited authority to remove AI-generated content

Stack Overflow’s moderators are on strike over a new policy that limits their ability to remove AI-generated content....

Generative AI holds promise for marketers, finds Salesforce study

Over 1,000 marketing professionals from all sectors and sizes are exploring the practical uses of generative AI while...

Altman tips Israel for big role in AI risk mitigation

During a discussion with Israeli President Isaac Herzog, OpenAI CEO Sam Altman voiced his opinion that Israel can... records two million downloads in first week

A new AI platform founded by ex-Google employees Noam Shazeer and Daniel De Freitas with over 10 million...

Become a member

New, Relevant Tech Stories. Our article selection is done by industry professionals. Our writers summarize them to give you the key takeaways