important part comes a little later. After some weeks or months, blind users begin performing well.34 But not just because they have memorized the translation. Instead, they are, in some sense, seeing. In a strange, low-resolution way, they are experiencing vision.35 One of the vOICe users, who went blind after twenty years of vision, had this to say about her experience of using the device:
You can develop a sense of the soundscapes within two to three weeks. Within three months or so you should start seeing the flashes of your environment where you’d be able to identify things just by looking at them. . . . It is sight. I know what sight looks like. I remember it.36
Rigorous training is key. Just as with cochlear implants, it can take many months of using these technologies before the brain starts to make sense of the signals. By that point, the changes are measurable in brain imaging. A particular region of the brain (the lateral occipital cortex) normally responds to shape information, whether the shape is determined by sight or by touch. After users wear the glasses for several days, this brain region becomes activated by the soundscape.37 The improvements in a user’s performance are paralleled by the amount of cerebral reorganization.38
In other words, the brain figures out how to extract shape information from incoming signals, regardless of the path by which those signals get into the inner sanctum of the skull—whether by sight, touch, or sound. The details of the detectors don’t matter. All that matters is the information they carry.
By the early years of the twenty-first century, several laboratories began to take advantage of cell phones, developing apps to convert camera input into audio output. Blind people listen through their earbuds as they view the scene in front of them with the cell phone camera. For example, the vOICe can now be downloaded for free on phones around the world.
The vOICe is not the only visual-to-auditory substitution approach; recent years have seen a proliferation of these technologies. For example, the EyeMusic app uses musical pitches to represent the up-down location of pixels: the higher a pixel, the higher the note. Timing is exploited to represent the left-right pixel location: earlier notes are used for something on the left; later notes represent something on the right. Color is conveyed by different musical instruments: white (vocals), blue (trumpet), red (organ), green (reed), yellow (violin).39 Other groups are experimenting with alternative versions: for example, using magnification at the center of the scene, just like the human eye, or using simulated echolocation or distance-dependent hum volume modulation, or many other ideas.40
The ubiquity of smartphones has moved the world from bulky computers to colossal power in the back pocket. And this allows not only efficiency and speed but also a chance for sensory-substitution devices to gain global leverage, especially as 87 percent of visually impaired people live in developing countries.41 Inexpensive sensory-substitution apps can have a worldwide reach, because they involve no ongoing cost of production, physical dissemination, stock replenishment, or adverse medical reactions. In this way, a neurally inspired approach can be inexpensive and rapidly deployable and tackle global health challenges.
If it seems surprising that a blind person can come to “see” with her tongue or through cell phone earbuds, just remember how the blind come to read Braille. At first the experience involves mysterious bumps on the fingertips. But soon it becomes more than that: the brain moves beyond the details of the medium (the bumps) for a direct experience of the meaning. The Braille reader’s experience parallels yours as your eyes flow over this text: although these letters are arbitrary shapes, you surpass the details of the medium (the letters) for a direct experience of the meaning.
To the first-time wearer of the tongue grid or the sonic headphones, the data streaming in require translation: the signals generated by a visual scene (say, a dog entering the living room with a bone in her mouth) give little indication of what’s out there. It’s as though the nerves were passing messages in a foreign language. But with enough practice, the brain can learn to translate. And once it does, the understanding of the visual world becomes directly apparent.
GOOD VIBRATIONS
Given that 5 percent of the world has disabling hearing loss, researchers some years ago got interested in ferreting out the genetics of that.42 Unfortunately, the community has currently discovered more than 220 genes associated with deafness. For those hoping for a simple solution, this is a disappointment, but it’s no surprise. After all, the auditory system works as a symphony of many delicate pieces operating in concert. As with any complex system, there are hundreds of ways it can be disrupted. As soon as any part of the system goes wrong, the whole system suffers, and the result is lumped into the term “hearing loss.”
Many researchers are working to figure out how to repair those individual pieces and parts. But let’s ask the question from the livewiring point of view: How could the principles of sensory substitution help us solve the problem?
With this question in mind, my former graduate student Scott Novich and I set out to build sensory substitution for the deaf. We set out to build something totally unobtrusive—so unobtrusive that no one would even know you had it. To that end, we assimilated several advances in high-performance computing into a sound-to-touch sensory-substitution device worn under the shirt. Our Neosensory Vest captures the sound around you and maps it onto vibratory motors on the skin. People can feel the sonic world around them.
If it sounds strange that this could work, just note that this is all your inner ear does: it breaks up sound into different frequencies (from low to high), and then those data get shipped off to the brain for interpretation. In essence, we’re just transferring the inner ear to the skin.
The Neosensory Vest. Sound is translated into patterns of vibration on the skin.
The skin is a mind-bogglingly sophisticated computational material, but we do not use it for much in modern life. It’s the kind of material you’d pay great sums for if it were synthesized in a Silicon Valley factory, but currently this material hides beneath your clothing, almost entirely unemployed. However, you may wonder whether the skin has enough bandwidth to transmit all the information of sound. After all, the cochlea is an exquisitely specialized structure, a masterpiece of capturing and encoding sound. The skin, in contrast, is focused on other measures, and it has poor spatial resolution. Conveying an inner ear’s worth of information to the skin would require several hundred vibrotactile motors—too many to fit on a person. But by compressing the speech information, we can use fewer than thirty motors. How? Compression is all about extracting the important information into the smallest description. Think about chatting on your cell phone: you speak and the other person hears your voice. But the signal representing your voice is not what is getting directly transmitted. Instead, the phone digitally samples your speech (takes a momentary record of it) eight thousand times each second. Algorithms then summarize the important elements from those thousands of measures, and the compressed signal is what gets sent to the cell phone tower. Leveraging these kinds of compression techniques, the Vest captures sounds and “plays” a compressed representation with multiple motors on the skin.43
Our first participant was a thirty-seven-year-old named Jonathan who was born profoundly deaf. We had Jonathan train with the Neo-sensory Vest for four days, two hours a day, learning from a set of thirty words. On the fifth day, Scott shields his mouth (so his lips can’t be read) and says the word “touch.” Jonathan feels the complicated pattern of vibrations on his torso. And then he writes the word “touch” on the dry-erase board. Scott now says a different word (“where”), and Jonathan writes it on the board. Jonathan is able to translate the complicated pattern of vibrations into an understanding of the word that’s spoken. He’s not doing the decoding consciously, because the patterns are too complicated; instead, his brain is unlocking the patterns. When we switched to a new set of words, his performance stayed high, indicating that he was not simply memorizing but learning how to hear. In other words, if you have normal hearing, I can say a