Scientists have unveiled Brain-IT, a groundbreaking artificial intelligence programme capable of reconstructing the exact images a person is looking at by analysing their brain scans.
The system was developed by researchers at the Weizmann Institute of Science. It recreates pictures that it has never seen before with remarkable accuracy, improving on previous models that struggled with basic features such as composition and colour.
During a recent study, participants were asked to look at various photographs, including images of a baseball game, a dog leaning out of a car and a group trekking across a snowy plain. The artificial intelligence analysed the brain activity patterns of the volunteers through a scan and produced highly accurate reproductions of the original photographs.
Professor Michal Irani, from the Weizmann Institute of Science, said there are models today that translate brain activity into images and produce impressive reconstructions that preserve the semantic meaning reasonably well.
"However, they tend to make mistakes in basic features such as composition and colour," Professor Irani said.
"The new model we developed outperforms them in reconstructing both the content of the image and its details. What's more, while every other model requires dozens of hours of brain scans to learn to 'read' a new person, our model needs only one hour."
The Weizmann Institute of Science is a major public research university located in Rehovot, Israel, and is known globally for its extensive scientific and multidisciplinary research.
Rapid learning from neural data
A significant advantage of the Brain-IT system is the speed at which it can learn to interpret the neural patterns of a new user. Other mind reading tools typically require about 40 hours of brain scan data from a new individual before they can successfully predict what that person is seeing.
The research team explained that their new decoder only needs 60 minutes of data to achieve similar results.
To prove this rapid learning capability, the scientists compared the images generated by Brain-IT when it was trained with just one hour of data against the outputs from the same system trained with 40 hours of data. The resulting image reconstructions were remarkably similar.
The team also compared the image recreation results of Brain-IT with those of other established programmes, revealing that the new system generated much more accurate reconstructions across the board.
To build Brain-IT, the scientists fed the system thousands of brain scans collected while eight volunteers looked at different images. This extensive dataset allowed the artificial intelligence to learn exactly how specific patterns of neural activity correspond to different colours, shapes and objects.
Mapping the human brain
The system became so accurate during its development that it even learned to predict what a human brain scan would look like based solely on an image. By combining data from multiple studies, the researchers successfully identified brain regions that appear to perform similar functions across different people.
The researchers noted that one specific area of the brain consistently responded to images of food, while another distinct area was activated primarily by pictures of sport.
"During training, the encoder naturally identified 128 functional regions that are shared by all people and perform specific roles in image processing," Professor Irani said.
"Some of them are familiar to neuroscientists, but others are entirely new."
The team discovered a new division of roles within the parahippocampal place area, a specific region of the brain that processes images of places. Professor Irani explained that one part of this region responds specifically to indoor scenes, while another distinct part responds to outdoor scenes.
When given a new brain scan, the artificial intelligence was able to generate a remarkably accurate reconstruction of the original image, sometimes referred to as a RECON by the researchers.
Future of cognitive decoding
Functional magnetic resonance imaging, the scanning technology typically used in these cognitive studies, measures brain activity by detecting changes associated with blood flow. When an area of the brain is in use, blood flow to that region increases.
Professor Irani and her laboratory are now working to extend these decoding methods beyond visual data to process auditory information.
The research team is also looking at the possibility of decoding video, which presents a significantly harder technical challenge due to the speed at which visual information changes.
Professor Irani explained that decoding video remains especially challenging, such as during dreaming.
"Dozens of images change every second, while an fMRI scan takes about two seconds. If we overcome all these obstacles, it's possible that in the future we may even be able to read dreams," she said.
Advances in sensor technology
The scientists are also working towards adapting similar systems to decode brain activity recorded using electroencephalography.
Electroencephalography is a monitoring method that measures the electrical signals of the brain through sensors placed directly on the scalp. The researchers noted this can sometimes be achieved via a specialised cap or even specially designed headphones.
As artificial intelligence models become increasingly sophisticated, scientists expect that it will become much easier to interpret brain data collected through these simpler wearable sensors rather than relying on large magnetic resonance imaging scanners.
The findings regarding the Brain-IT system were presented at the Cognitive Computational Neuroscience conference in New York last month.
