Scientists have trained a computer to analyze the brain activity of someone listening to music and based on these neural patterns alone, to plays the song.
The research, which published last Tuesday, produced a recognizable version of the Pink Floyd song (from 1979), “Another Brick in the Wall (Part 1)”.
To collect the data for the study, researchers imaged the brains of 29 epilepsy patients at Albany Medical Center in New York state from 2009 to 2015. As part of their epilepsy treatment, the patients had a net of "claws" electrodes implanted in their brains.
This created a rare opportunity for neuroscientists to record their brain activity while listening to music. The team chose the Pink Floyd song in part because the older patients liked it. "If they said, 'I can't listen to this rubbish,' then the data would be terrible," said Dr. Schalk. In addition, the song features 41 seconds of lyrics and two and a half minutes of looping instrumentals, a combination that was useful in showing how the brain processes words as well as melody.
Robert Knight, a neuroscientist at the University of California (Berkeley), and the leader of the team, asked one of his postdocs, Ludovic Bellier, to try to use the data set to reconstruct the music. The lab had already done similar work reconstructing words.
Analyzing the data from each patient, Dr. Bellier identified which parts of the brain lit up during the song and what frequencies those areas responded to. Just as the resolution of an image depends on its number of pixels, the quality of an audio recording depends on the number of frequencies it can represent.
To reconstruct “Another Brick in the Wall”, the researchers used 128 frequency bands. This meant training 128 computer models centered around the song. The researchers then tested the effect on four individual brains through the model. The resulting results were all recognizable and related to the Pink Floyd song, albeit with notable differences. The placement of electrodes from patient to patient likely explains the variations, the researchers said, but personal characteristics, such as whether a person was a musician, also matter.
Note that this particular approach was limited: the scientists could only see brain activity where doctors had placed electrodes to look for seizures.
The researchers also discovered a spot in the temporal lobe of the brain that reacted when volunteers heard the song's 16 guitar notes. So they suggested that this particular region might be involved in our perception of rhythm.
The above findings offer a first step towards creating more expressive devices to assist people who cannot speak. In recent years, scientists have made major breakthroughs in extracting words from electrical signals produced by the brains of people with muscular paralysis when they try to speak.
