August 18, 2013

Deep Learning


Deep Learning


When Ray Kurzweil met with Google CEO Larry Page last July, he wasn’t looking for a job. A respected inventor who’s become a machine-intelligence futurist, Kurzweil wanted to discuss his upcoming book How to Create a Mind. He told Page, who had read an early draft, that he wanted to start a company to develop his ideas about how to build a truly intelligent computer: one that could understand language and then make inferences and decisions on its own.

It quickly became obvious that such an effort would require nothing less than Google-scale data and computing power. “I could try to give you some access to it,” Page told Kurzweil. “But it’s going to be very difficult to do that for an independent company.” So Page suggested that Kurzweil, who had never held a job anywhere but his own companies, join Google instead. It didn’t take Kurzweil long to make up his mind: in January he started working for Google as a director of engineering. “This is the culmination of literally 50 years of my focus on artificial intelligence,” he says.

Kurzweil was attracted not just by Google’s computing resources but also by the startling progress the company has made in a branch of AI called deep learning. Deep-learning software attempts to mimic the activity in layers of neurons in the neocortex, the wrinkly 80 percent of the brain where thinking occurs. The software learns, in a very real sense, to recognize patterns in digital representations of sounds, images, and other data.

The basic idea—that software can simulate the neocortex’s large array of neurons in an artificial “neural network”—is decades old, and it has led to as many disappointments as breakthroughs. But because of improvements in mathematical formulas and increasingly powerful computers, computer scientists can now model many more layers of virtual neurons than ever before.

With this greater depth, they are producing remarkable advances in speech and image recognition. Last June, a Google deep-learning system that had been shown 10 million images from YouTube videos proved almost twice as good as any previous image recognition effort at identifying objects such as cats. Google also used the technology to cut the error rate on speech recognition in its latest Android mobile software. In October, Microsoft chief research officer Rick Rashid wowed attendees at a lecture in China with a demonstration of speech software that transcribed his spoken words into English text with an error rate of 7 percent, translated them into Chinese-language text, and then simulated his own voice uttering them in Mandarin. That same month, a team of three graduate students and two professors won a contest held by Merck to identify molecules that could lead to new drugs. The group used deep learning to zero in on the molecules most likely to bind to their targets.

Google in particular has become a magnet for deep learning and related AI talent. In March the company bought a startup cofounded by Geoffrey Hinton, a University of Toronto computer science professor who was part of the team that won the Merck contest. Hinton, who will split his time between the university and Google, says he plans to “take ideas out of this field and apply them to real problems” such as image recognition, search, and natural-language understanding, he says.

All this has normally cautious AI researchers hopeful that intelligent machines may finally escape the pages of science fiction. Indeed, machine intelligence is starting to transform everything from communications and computing to medicine, manufacturing, and transportation. The possibilities are apparent in IBM’s Jeopardy!-winning Watson computer, which uses some deep-learning techniques and is now being trained to help doctors make better decisions. Microsoft has deployed deep learning in its Windows Phone and Bing voice search.

Extending deep learning into applications beyond speech and image recognition will require more conceptual and software breakthroughs, not to mention many more advances in processing power. And we probably won’t see machines we all agree can think for themselves for years, perhaps decades—if ever. But for now, says Peter Lee, head of Microsoft Research USA, “deep learning has reignited some of the grand challenges in artificial intelligence.”

Building a Brain


There have been many competing approaches to those challenges. One has been to feed computers with information and rules about the world, which required programmers to laboriously write software that is familiar with the attributes of, say, an edge or a sound. That took lots of time and still left the systems unable to deal with ambiguous data; they were limited to narrow, controlled applications such as phone menu systems that ask you to make queries by saying specific words.

Neural networks, developed in the 1950s not long after the dawn of AI research, looked promising because they attempted to simulate the way the brain worked, though in greatly simplified form. A program maps out a set of virtual neurons and then assigns random numerical values, or “weights,” to connections between them. These weights determine how each simulated neuron responds—with a mathematical output between 0 and 1—to a digitized feature such as an edge or a shade of blue in an image, or a particular energy level at one frequency in a phoneme, the individual unit of sound in spoken syllables.


Some of today’s artificial neural networks can train themselves to recognize complex patterns.


Programmers would train a neural network to detect an object or phoneme by blitzing the network with digitized versions of images containing those objects or sound waves containing those phonemes. If the network didn’t accurately recognize a particular pattern, an algorithm would adjust the weights. The eventual goal of this training was to get the network to consistently recognize the patterns in speech or sets of images that we humans know as, say, the phoneme “d” or the image of a dog. This is much the same way a child learns what a dog is by noticing the details of head shape, behavior, and the like in furry, barking animals that other people call dogs.

But early neural networks could simulate only a very limited number of neurons at once, so they could not recognize patterns of great complexity. They languished through the 1970s.


WHY IT MATTERS

Computers would assist humans far more effectively if they could reliably recognize patterns and make inferences about the world.

Breakthrough
A method of artificial intelligence that could be generalizable to many kinds of applications.

Key Players
• Google
• Microsoft
• IBM
• Geoffrey Hinton, University of Toronto

August 17, 2013

Smart Watches


Pebble Smart Watch
Pebble Smart Watches


Eric Migicovsky didn’t really want a “wearable computer.” When he first conceived of what would become the Pebble smart watch five years ago, as an industrial-design student at Delft University of Technology in the Nether­lands, he just wanted a way to use his smartphone without crashing his bicycle. “I thought of creating a watch that could grab information from my phone,” the 26-year-old Canadian says. “I ended up building a prototype in my dorm room.”

Now Migicovsky is shipping 85,000 Pebble watches to eager customers who don’t want to lug a glass slab out of their pocket just to check their e-mail or the weather forecast. Pebble uses Bluetooth to connect wirelessly to an iPhone or Android phone and displays notifications, messages, and other simple data of the user’s choosing on its small black-and-white LCD screen. In April 2012, using the online fund-raising platform Kickstarter, Migicovsky asked for $100,000 to help bring Pebble to market. Five weeks later, he had more than $10 million—making his the highest-grossing Kickstarter campaign yet. Suddenly smart watches are a real product category: Sony entered the market last year,Samsung is about to, and Apple seems likely to follow.




Although the $150 Pebble watch can be used to control a music playlist or run simple apps like RunKeeper, a cloud-based fitness tracker, Migicovsky and his team purposely designed the watch to do as little as possible, leaving more complicated apps for phones. This emphasis on making the watch “glanceable” informed nearly every aspect of the design. The black-and-white screen, for example, can be read in direct sunlight and displays content persistently without needing to “sleep” to conserve battery power, as color or touch-screen displays do.


These watches are coming to market a few months before Google Glass, which is another attempt to solve the problem Pebble addresses—namely, that “interacting with our phones has a certain overhead that doesn’t need to be there,” says Mark Rolston, chief creative officer of Frog Design. But Google Glass will try to replace the smartphone altogether by combining a computer and monitor into eyeglass frames so that wearers can “augment” their view of the world with data. That lines up with predictions about the advent of wearable computing, but it’s easy to see Pebble’s idea being much more popular. By making use of a watch—a classic accessory—Pebble is trying to fit in to long-standing social norms rather than create new ones.

Why It Matters

Even as computing gets more sophisticated, people want simple and easy-to-use interfaces.

Breakthrough
Watches that pull selected data from mobile phones so their wearers can absorb information with a mere glance.

Key Players
• Pebble
• Sony
• Motorola
• MetaWatch

August 10, 2013

Ultra-Efficient Solar Power



Harry Atwater thinks his lab can make an affordable device that produces more than twice the solar power generated by today’s panels. The feat is possible, says the Caltech professor of materials science and applied physics, because of recent advances in the ability to manipulate light at a very small scale.

Solar panels on the market today consist of cells made from a single semiconducting material, usually silicon. Since the material absorbs only a narrow band of the solar spectrum, much of sunlight’s energy is lost as heat: these panels typically convert less than 20 percent of that energy into electricity. But the device that ­Atwater and his colleagues have in mind would have an efficiency of at least 50 percent. It would use a design that efficiently splits sunlight, as a prism does, into six to eight component wavelengths—each one of which produces a different color of light. Each color would then be dispersed to a cell made of a semiconductor that can absorb it.






Atwater’s team is working on three designs. In one (see illustration), for which the group has made a prototype, sunlight is collected by a reflective metal trough and directed at a specific angle into a structure made of a transparent insulating material. Coating the outside of the transparent structure are multiple solar cells, each made from one of six to eight different semiconductors. Once light enters the material, it encounters a series of thin optical filters. Each one allows a single color to pass through to illuminate a cell that can absorb it; the remaining colors are reflected toward other filters designed to let them through.

Solar Power Grid
Solar Power Grid


Another design would employ nanoscale optical filters that could filter light coming from all angles. And a third would use a hologram instead of filters to split the spectrum. While the designs are different, the basic idea is the same: combine conventionally designed cells with optical techniques to efficiently harness sunlight’s broad spectrum and waste much less of its energy.

It’s not yet clear which design will offer the best performance, says Atwater. But the devices envisioned would be less complex than many electronics on the market today, he says, which makes him confident that once a compelling prototype is fabricated and optimized, it could be commercialized in a practical way.




Achieving ultrahigh efficiency in solar designs should be a primary goal of the industry, argues Atwater, since it’s now “the best lever we have” for reducing the cost of solar power. That’s because prices for solar panels have plummeted over the past few years, so continuing to focus on making them less expensive would have little impact on the overall cost of a solar power system; expenses related to things like wiring, land, permitting, and labor now make up the vast majority of that cost. Making modules more efficient would mean that fewer panels would be needed to produce the same amount of power, so the costs of hardware and installation could be greatly reduced. “Within a few years,” Atwater says, “there won’t be any point to working on technology that has efficiency that’s less than 20 percent.”


Why It Matters


Higher efficiency would make solar power more competitive with fossil fuels.

Breakthrough
Managing light to harness more of sunlight’s energy.


Key Players
• Harry Atwater, Caltech
• Albert Polman, AMOLF
• Eli Yablonovitch,
University of
California, Berkeley
• Dow Chemical

August 3, 2013

Memory Implants



Theodore Berger, a biomedical engineer and neuroscientist at the University of Southern California in Los Angeles, envisions a day in the not too distant future when a patient with severe memory loss can get help from an electronic implant. In people whose brains have suffered damage from Alzheimer’s, stroke, or injury, disrupted neuronal networks often prevent long-term memories from forming. For more than two decades, Berger has designed silicon chips to mimic the signal processing that those neurons do when they’re functioning properly—the work that allows us to recall experiences and knowledge for more than a minute. Ultimately, Berger wants to restore the ability to create long-term memories by implanting chips like these in the brain.

The idea is so audacious and so far outside the mainstream of neuroscience that many of his colleagues, says Berger, think of him as being just this side of crazy. “They told me I was nuts a long time ago,” he says with a laugh, sitting in a conference room that abuts one of his labs. But given the success of recent experiments carried out by his group and several close collaborators, Berger is shedding the loony label and increasingly taking on the role of a visionary pioneer.



Berger and his research partners have yet to conduct human tests of their neural prostheses, but their experiments show how a silicon chip externally connected to rat and monkey brains by electrodes can process information just like actual neurons. “We’re not putting individual memories back into the brain,” he says. “We’re putting in the capacity to generate memories.” In an impressive experiment published last fall, Berger and his coworkers demonstrated that they could also help monkeys retrieve long-term memories from a part of the brain that stores them.

If a memory implant sounds farfetched, Berger points to other recent successes in neuroprosthetics. Cochlear implants now help more than 200,000 deaf people hear by converting sound into electrical signals and sending them to the auditory nerve. Meanwhile, early experiments have shown that implanted electrodes can allow paralyzed people to move robotic arms with their thoughts. Other researchers have had preliminary success with artificial retinas in blind people.

Still, restoring a form of cognition in the brain is far more difficult than any of those achievements. Berger has spent much of the past 35 years trying to understand fundamental questions about the behavior of neurons in the hippocampus, a part of the brain known to be involved in forming memory. “It’s very clear,” he says. “The hippocampus makes short-term memories into long-term memories.”
What has been anything but clear is how the hippocampus accomplishes this complicated feat. Berger has developed mathematical theorems that describe how electrical signals move through the neurons of the hippocampus to form a long-term memory, and he has proved that his equations match reality. “You don’t have to do everything the brain does, but can you mimic at least some of the things the real brain does?” he asks. “Can you model it and put it into a device? Can you get that device to work in any brain? It’s those three things that lead people to think I’m crazy. They just think it’s too hard.”


Cracking the Code

Berger often speaks in sentences that stretch to paragraph length and have many asides, footnotes, and complete diversions from the point. I ask him to define memory. “It’s a series of electrical pulses over time that are generated by a given number of neurons,” he says. “That’s important because you can reduce it to this and put it back into a framework. Not only can you understand it in terms of the biological events that happened; that means that you can poke it, you can deal with it, you can put an electrode in there, and you can record something that matches your definition of a memory. You can find the 2,147 neurons that are part of this memory. And what do they generate? They generate this series of pulses. It’s not bizarre. It’s something you can handle. It’s useful. It’s what happens.”
This is the conventional view of memory, but it only scratches the surface. And to Berger’s perpetual frustration, many colleagues who probe this mysterious realm of the brain haven’t attempted to go much deeper. Neuroscientists track electrical signals in the brain by monitoring action potentials, microvolt changes on the surfaces of neurons. But all too often, says Berger, their reports oversimplify what’s actually taking place. “They find an important event in the environment and count action potentials,” he says. “They say, ‘It went up from 1 to 200 after I did something. I’m finding something interesting.’ What are you finding? ‘Activity went up.’ But what are you finding? ‘Activity went up.’ So what? Is it coding something? Is it representing something that the next neuron cares about? Does it make the next neuron do something different? That’s what we’re supposed to be doing: explaining things, not just describing things.”
If one neuron fires at a specific time and place, what exactly do the neighboring neurons do in response?
Berger takes a marker and fills a whiteboard from top to bottom with a line of circles that represent neurons. Next to each one, he draws a horizontal line that has a different pattern of blips on it. “This is you in my brain,” he says. “My hippocampus has already formed a long-term memory of you. I’ll remember you into next week. But how can I distinguish you from the next person? Let’s say there are 500,000 cells in the hippocampus that represent you, and there are all sorts of things that each cell is coding—like how your nose is relative to your eyebrow—and they code that with different patterns. So the reality of the nervous system is really complicated, which is why we’re still asking such basic, limited questions about it.”


Theodore Berger has spent his career trying to understand how neurons form memories.

In graduate school at Harvard, ­Berger’s mentor was Richard Thompson, who studied localized, learning-induced changes in the brain. Thompson used a tone and a puff of air to condition rabbits to blink their eyes, aiming to determine where the memory he induced was stored. The idea was to find a specific place in the brain where the learning was localized, says Berger: “If the animal did learn and you removed it, the animal couldn’t remember.”

Thompson, with Berger’s help, managed to do just that; they published the results in 1976. To find the site in the rabbits, they equipped the animals’ brains with electrodes that could monitor the activity of a neuron. Neurons have gates on their membranes, which let electrically charged particles like sodium and potassium in and out. Thompson and Berger documented the electrical spikes seen in the hippocampus as rabbits developed a memory. Both the spikes’ amplitude (representing the action potential) and their spacing formed patterns. It can’t be an accident, Berger thought, that cells fire in a way that forms patterns with respect to time.

This led him to a central question that underlies his current work: as cells receive and send electrical signals, what pattern describes the quantitative relationship between the input and the output? That is, if one neuron fires at a specific time and place, what exactly do the neighboring neurons do in response? The answer could reveal the code that neurons use to form a long-term memory.

But it soon became clear that the answer was extremely complex. In the late 1980s, Berger, working at the University of Pittsburgh with Robert Sclabassi, became fascinated by a property of the neuronal network in the hippocampus. When they stimulated the hippocampus of a rabbit with electrical pulses (the input) and charted how signals moved through different populations of neurons (the output), the relationship they observed between the two wasn’t linear. “Let’s say you put in 1 and get 2,” says Berger. “That’s pretty easy. It’s a linear relation.” It turns out, however, that there’s “essentially no condition in the brain where you get linear activity, a linear summation,” he says. “It’s always nonlinear.” Signals overlap, with some suppressing an incoming pulse and some accentuating it.

By the early 1990s, his understanding—and computing hardware—had advanced to the point that he could work with his colleagues at the University of Southern California’s department of engineering to make computer chips that mimic the signal processing done in parts of the hippocampus. “It became obvious that if I could get this stuff to work in large numbers in hardware, you’ve got part of the brain,” he says. “Why not hook up to what’s existing in the brain? So I started thinking seriously about prosthetics long before anybody even considered it.”


A Brain Implant

Berger began working with Vasilis ­Marmarelis, a biomedical engineer at USC, to begin making a brain prosthesis (see “Regaining Lost Brain Function”). They first worked with hippocampal slices from rats. Knowing that neuronal signals move from one end of the hippocampus to the other, the researchers sent random pulses into the hippocampus, recorded the signals at various locales to see how they were transformed, and then derived mathematical equations describing the transformations. They implemented those equations in computer chips.

Next, to assess whether such a chip could serve as a prosthesis for a damage hippocampal region, the researchers investigated whether they could bypass a central component of the pathway in the brain slices. Electrodes placed in the region carried electrical pulses to an external chip, which performed the transformations normally done in the hippocampus. Other electrodes delivered the signals back to the slice of brain.
“I never thought I’d see this go into humans, and now our discussions are about when and how. I never thought I’d live to see the day.”
Then the researchers took a leap forward by trying this in live rats, showing that a computer could in fact serve as an artificial component of the hippocampus. They began by training the animals to push one of two levers to receive a treat, recording the series of pulses in the hippocampus as they chose the correct one. Using those data, Berger and his team modeled the way the signals were transformed as the lesson was converted into a long-term memory, and they captured the code believed to represent the memory itself. They proved that their device could generate this long-term memory code from input signals recorded in rats’ brains while they learned the task. Then they gave the rats a drug that interfered with their ability to form long-term memories, causing them to forget which lever produced the treat. When the researchers pulsed the drugged rats’ brains with the code, the animals were again able to choose the right lever.

Last year, the scientists published primate experiments involving the prefrontal cortex, a part of the brain that retrieves the long-term memories created by the hippocampus. They placed electrodes in the monkey brains to capture the code formed in the prefrontal cortex that they believed allowed the animals to remember an image they had been shown earlier. Then they drugged the monkeys with cocaine, which impairs that part of the brain. Using the implanted electrodes to send the correct code to the monkeys’ prefrontal cortex, the researchers significantly improved the animal’s performance on the image-identification task.

Within the next two years, Berger and his colleagues hope to implant an actual memory prosthesis in animals. They also want to show that their hippocampal chips can form long-term memories in many different behavioral situations. These chips, after all, rely on mathematical equations derived from the researchers’ own experiments. It could be that the researchers were simply figuring out the codes associated with those specific tasks. What if these codes are not generalizable, and different inputs are processed in various ways? In other words, it is possible that they haven’t cracked the code but have merely deciphered a few simple messages.

Berger allows that this may well be the case, and his chips may form long-term memories in only a limited number of situations. But he notes that the morphology and biophysics of the brain constrain what it can do: in practice, there are only so many ways that electrical signals in the hippocampus can be transformed. “I do think we’re going to find a model that’s pretty good for a lot of conditions and maybe most conditions,” he says. “The goal is to improve the quality of life for somebody who has a severe memory deficit. If I can give them the ability to form new long-term memories for half the conditions that most people live in, I’ll be happy as hell, and so will be most patients.”

Despite the uncertainties, Berger and his colleagues are planning human studies. He is collaborating with clinicians at his university who are testing the use of electrodes implanted on each side of the hippocampus to detect and prevent seizures in patients with severe epilepsy. If the project moves forward as envisioned, Berger’s group will piggyback on the trial to look for memory codes in those patients’ brains.

“I never thought I’d see this go into humans, and now our discussions are about when and how,” he says. “I never thought I’d live to see the day, but now I think I will.”



WHY IT MATTERS

Brain damage can cause people to lose the ability to form long-term memories.

Breakthrough
Animal experiments show it is possible to correct for memory problems with implanted electrodes.

Key Players
• Theodore Berger, USC
• Sam Deadwyler, Wake Forest
• Greg Gerhardt,
University of Kentucky
• DARPA