Watson on Jeopardy!

• Posted 01.20.11
• NOVA

A computer taking on the best human contestants on Jeopardy!?—the prospect is perhaps even more astonishing than when IBM's Deep Blue computer challenged the world's top chess player in 1997 (and won). NOVA recently asked three experts on artificial intelligence about Watson, its capabilities, and its implications beyond the game show. They include David Ferrucci, head of the IBM team that programmed Watson to play on Jeopardy!; Rodney Brooks, roboticist and recently retired head of the Computer Science and Artificial Intelligence Laboratory at MIT; and Luis von Ahn, a computer scientist at Carnegie Mellon and an inventor of CAPTCHA, the squiggly words and numbers used in computing to ensure a response is not coming from a computer. The following is an edited version of that conversation.

WATSON AND JEOPARDY!

How is Watson so successful with understanding natural language?

David Ferrucci (Research Staff Member and leader of the Semantic Analysis and Integration Department at IBM's Thomas J. Watson Research Center): I think what's making Watson successful is its internal architecture. It's looking at so many different algorithms—thousands of different algorithms—some of them focused on understanding the question, weighting the various terms, looking at the grammar, the syntax, finding the phrases, the keywords, the entities, the dates, the times, trying to understand what it is being asked. And this, in itself, is a big challenge, where we use a variety of different technologies. But ultimately, what's exciting about it is how it looks at many, many different possibilities and assesses them and builds confidence in a final answer to decide whether or not it's correct and whether or not it wants to risk buzzing in on Jeopardy!

Why did you guys choose Jeopardy!? What about this game show in particular makes it the ideal challenge for a computer like Watson?

Ferrucci: Well, Jeopardy!'s a really fascinating game that challenges a computer to deal with language, to look at an unusually broad domain. You can't anticipate ahead of time that there will be a question about a particular single topic, for example, or that it's going to be phrased any one way. So you have that breadth. You also have to have precision. You have to have the right answer in the top spot. No points for second or third place or somewhere in the top 10 documents. You have to have that confidence. You have to know that you've got it right, otherwise you don't want to risk buzzing in, and you have to do it really, really quickly. So all of these aspects of the game help us push that natural-language-understanding technology in a way that we've never really been able to do before.

What happens if Watson crashes during the taping of Jeopardy!? What will you guys do?

Ferrucci: Well, we cut tape—is that the expression they use?—and we see if we can bring it back up. But Watson is complicated. It's essentially 10 refrigerators' worth of hardware. There's about a million lines of new code in there. So it's a complex system. We've done a tremendous amount of testing, but anybody who designs software knows something could always go wrong. And we're prepared to fix it and bring it back up rapidly. We'll see.

WATSON'S SIGNIFICANCE

So what implications does Watson have beyond competing on Jeopardy!?

Ferrucci: We looked at Jeopardy! as a challenge that drives the technology, not, obviously, as an end goal. And we kept a careful eye to focus on a reasonable capability, something that ultimately will give us the ability to look at huge volumes of text, do a better job at understanding them, and pull out the information that humans are looking for, the precise facts, the precise opinions, whatever it is that you're asking about, to do a deeper understanding of what you want and being able to get that breadth and that precision over language to get you the right answers. And we've already seen that helping. We've already started looking at applying this technology to a number of different areas, including medicine and healthcare as well as text support, publishing, finance. We're actually very excited about some of the preliminary results.

"Whether it wins or not, to me, is irrelevant. It's being able to be in the game that is the really big advance."

What about for AI and computer science? What is Watson's promise for those?

Luis von Ahn (Professor of Computer Science at Carnegie Mellon University): I think there are a lot of implications, not just for AI and computer science but also for the world, for things like Google. Right now on Google, all you do is you type in some keywords, and it gives you links, but imagine if it could start answering your questions, as opposed to you having to go and find the answers. It's starting to do that. For example, if you go to Google and ask it, "What time is it in Austria?" it will tell you, but those are very simple questions. The implications of this is that it can give answers to much more complicated questions. I mean, it's probably not doing this yet, but you ask it things like "My head and my feet hurt—what do I have?" and it may just give you something right.

Rodney Brooks (Founder and Chief Technology Officer of Heartland Robotics and former Director of the Computer Science and Artificial Intelligence Laboratory at MIT): I think one of the really interesting things here is, because of the way Jeopardy! is set up, it really relies on wordplay and subtlety. It's forcing the IBM people to take that into account. Now, we use subtleties in our language all the time. We're just not aware of them because we're so good at it and so natural.

In one of the examples of Watson playing Jeopardy! was the clue, "A garment a small girl would wear on an operatic ship." And the answer was "pinafore." That pulled in so many different pieces of knowledge in just three seconds. What search would you use on Google to get that right now? You'd have to have all the thinking in your head. This is making the computer do it.

Ferrucci: One of the other questions I like is, "If you're standing, what direction do you look to see your wainscoting?" This sounds like such a simple question, but it's really fascinating what you have to do to answer that. I mean, humans sort of have to be situated in the world. They have to have some experience in interior design. They have to know what it means to be standing, what relative directions are. It's just really dramatic to think about what humans do to carefully understand and disambiguate and get at precise answers with language.

IMPLICATIONS FOR US

What if Watson wins? What would that say about us humans?

Brooks: Well, from someone who's not involved with building this particular machine, whether it wins or not, to me, is irrelevant. It's being able to be in the game that is the really big advance. Secondly, just because we've built something that can do new things that we couldn't do before doesn't mean there's lots of other things the computer can't do yet. But that it can do these things now, I think will help us tremendously in building search engines and interfaces.

And, you know, Garry Kasparov got beaten by another IBM machine—he was the world chess champion—and people are still playing chess. So I don't think it will stop us.

Von Ahn: Yeah, I have a very similar answer. I think it would be a tremendous achievement to win, but probably the bigger achievement is the fact that Watson is already competitive against some of the top players in the world, and it's definitely not going to put us out of a job just yet, but maybe later.

"Sometimes the computer is really sure it knows the answer and wants to be very aggressive with the buzzer. Other times it's not so sure."

What do you say to that, Dave? Are you guys building a machine that will take over using intelligence from humans?

Ferrucci: No. I think what this challenge helps us appreciate, frankly, is how incredible the human brain is. And it helps illuminate what's really hard for computers and what humans find natural and what we're looking for in terms of the right sort of human-machine interface. Wouldn't it be great to be able to communicate with the computer like Captain Picard or Captain Kirk does on "Star Trek," where you can fluently dialogue with an information-seeking computer that can understand what you're asking, ask follow up questions, and get exactly at the information that you need? That would be incredible. That's kind of this motivating vision, and whether Watson loses or not in this big game is really not the point. The point is we were able to take a step forward in that direction, and I think that's what we're most excited about.

A SELF-CONFIDENT COMPUTER

Playing successfully on Jeopardy! often involves ego and the confidence to gamble. Did you program self-confidence into Watson? How close are computers to breaking that barrier of human-like emotions?

Ferrucci: That's a great question. It's a fascinating one really. And one of the big challenges—and this is where we exploited machine learning in a big way—was computing that confidence and figuring out how to use that confidence to manage risk during a game.

So, for example, sometimes the computer is really sure it knows the answer and wants to be very aggressive with the buzzer. Other times it's not so sure, and it actually weighs how good its competitors are. Other times it feels its way ahead and doesn't want to take a risk, so it needs to be a lot more confident to buzz in. Sometimes it's desperate and actually wants to take a risk, even if it's not as sure. All that's in there, believe it or not. And you want to call those emotions. They're really not emotions. They're complex mathematical equations that we've trained into Watson over many, many simulations. It makes it a very fascinating challenge.

The other thing I'll mention about emotions, though, is Watson doesn't sweat! I sweat, but Watson doesn't. So we've seen games where Watson lost a big daily double and went down to zero and just kept right on going. Personally I would have fainted. [laughter]

Brooks: Interestingly, when Garry Kasparov was beaten by Deep Blue many years ago, he said, "Well, at least it didn't enjoy beating me." [laughter]

So we, as humans, like to hold onto what we've still got that the computers don't have. And what's happened here is there's an extra piece. You know, wordplay and puns and stuff like that used to be something that people could have but computers could never understand. Dave and his team now have them understanding it.

Ferrucci: I don't think you should worry. I mean, think about it this way: A computer is understanding language the way you might understand another language you don't know. Pick a language you don't know and then think about it. The only way you could understand it is by reading dictionaries in that language. You don't really connect those words to your experiences. They're not connected to your emotions. They're just connected to one another. The computer is using statistics. It doesn't actually enjoy or love or appreciate any of what those words represent.

Brooks: Yet. [laughter]

WHERE WE STAND

Yeah, are we on the verge of "Skynet," the artificially intelligent system in the Terminator movies?

Brooks: We knew Skynet was going to come up! [laughter] You know, one of the things people worry about is, as computers get smarter, are they going to replace us? Well, you are still here. Computers didn't replace you. You use them as tools. And this is getting better tools. And I suspect that's IBM's agenda here—they want to build better tools for people. I don't think we need to worry anytime soon about the machines taking over. I work in robotics, and the robots we build haven't gotten rid of people. They just make them more productive. We can relax for a few hundred years, is my guess.

"There are simple things that three-year-olds can do that computers cannot yet do."

Von Ahn: I think a few hundred years is a good answer. There are the very, very simple things that computers still cannot do. Even determining who somebody is from an image or whether something is a cat or a dog from an image is something that computers cannot do very well. So there are simple things that three-year-olds can do that computers cannot yet do.

So we don't have to worry about HAL from 2001...

Von Ahn: We don't have to worry about HAL just yet. Maybe in a hundred years or so. But that's okay; we'll be dead. [laughter]

One last question for Dave: What's the silliest answer you ever heard from Watson?

Ferrucci: Well, as Watson's developed over the years, it's had a lot of silly answers; there's quite a variety of them. I guess one of my favorites is we asked it "What do grasshoppers eat?" and its answer was "kosher." [laughter]