Watson @ Jeopardy Feb 14-16

[Qalbert]

that is precisely the problem, they didn't plan for the scenario. some audio recorder converting to text or something, idk, I'm not super tech inclined, but I'm sure there's a workaround if they had thought to plan it

[Zeb]

I missed the show earlier, can someone tell me how Watson was getting the questions anyways? The fairest way would be having visual or audio input, so Watson gets the question at the same time as the other two.

[Maguspk]

I wouldn't say they didn't plan for the scenario, that's a bit presumptuous to think something so trivial was overlooked. Any decent tester would consider such an edge case when drafting test cases for such a scenario of a Jeopardy game, and IBM has no shortage of 'decent' testers, they are a highly respected company as far as computer science knowledge and innovation goes. Analyzing and utilizing audio is simply outside the scope of the problem they are attempting to solve. You also open up a slew of potential avenues for error and inefficiency if you even consider other answers given orally by other contestants. For one you have to process the audio which then needs interpretation in the context of the question and then applied to your array of potential answers, all of which will delay your buzz trigger. This is all assuming you even interpret the audio correctly.

[Tyche]

I missed the show earlier, can someone tell me how Watson was getting the questions anyways? The fairest way would be having visual or audio input, so Watson gets the question at the same time as the other two.

They said he's fed a text file at the same time the others are read the question

[Qalbert]

Repeating a wrong answer that was just given is something a normal contestant wouldn't do, and as their goal was creating a machine capable of competing, it is something that should have been dealt with.

[Maguspk]

As viewers we see the question blown up but does the set of monitors displaying the board shown a blown up version of the question to the contestants or are they forced to listen to the question if they can't read it clearly from the stand? I wonder because if they do have to wait for the question to be read then it might be 'unfair' if Watson gets the question after it's revealed. I'm not sure how the buzzer system for Jeopardy works, obviously you can't buzz until Trebec is done speaking so there is the possibility Watson does not get the question until Trebec is done speaking but I highly doubt that. I'm sure the question is blown up for them too or something.

Repeating a wrong answer that was just given is something a normal contestant wouldn't do, and as their goal was creating a machine capable of competing, it is something that should have been dealt with.

The goal of Watson is to derive meaningful answers to questions posed in natural language via computation, Jeopardy is just an application of this capability. It wasn't built to win Jeopardy, they just believe it can.

[Skidz]

The goal of Watson is to derive meaningful answers to questions posed in natural language via computation, Jeopardy is just an application of this capability. It wasn't built to win Jeopardy, they just believe it can.

But it was obviously programmed to compete in jeopardy, (answering in the form of a question and all).

Not sure why you're trying so hard to defend watson man, lol. This should've come up in the "hundreds of test sessions watson went through to prepare for jeopardy".

[Blarg]

They didn't invest millions just to play and win at jeopardy. I agree with Magus here, competing in jeopardy is just an interesting thing that Watson can do but it is not the real purpose.

I am curious how watson knows when to buzz in. What data is given to him to let him know that trebec has stopped talking and how he would know when to buzz if another contestant beats him on the first buzz in but gets it wrong.

[Maguspk]

But it was obviously programmed to compete in jeopardy, (answering in the form of a question and all).

Not sure why you're trying so hard to defend watson man, lol. This should've come up in the "hundreds of test sessions watson went through to prepare for jeopardy".

I will try to state it simply, the cornerstone case where another's answer could be used to help him with his own answer is too far removed and specific to Jeopardy to be applicable to the technology driving Watson.

Watson was not trained to play Jeopardy, he was tested using Jeopardy questions. There is a difference. The only thing Watson does that is Jeopardy specific is prepending 'What is' to every answer. He doesn't even switch to other pronouns like 'who' where applicable, or use articles where necessary to form a proper sentence. He literally just throws 'what is' to the front to make his answer comply with the Jeopardy format. It's evident there were no major changes to make him work with Jeopardy and doing anything of the sort would be outside the scope of the research.

[Blarg]

Watson doesn't even have a robotic hand to write down his answer for final jeopardy. What a stupid waste of time Watson was, I can't believe they overlooked that.

[Skidz]

Watson went through "thousands of hours of testing and over fifty champion-level sparring matches" (http://www-03.ibm.com/innovation/us/watson/)

"Watson was optimized to tackle a specific challange: competing against the worlds best Jeopardy! contestants. Beyond Jeopardy!, the IBM team is working to deploy this technology across other industries" (http://www-03.ibm.com/innovation/us/...son/index.html)

And they did make changes to it so it can "play" Jeopardy, like installing a light sensor so it knows when to buzz in. I'm sure they could've come up with a solution to the problem.

[Maguspk]

Right so there are essential things it needs in order to even play Jeopardy 'on it's own, like it's buzz mechanism. These things do not include tangential considerations that affect how the algorithms work, the meat of Watson. Adding 'what is', detecting light, parsing sound of Trebec's voice, buzzing in, these are all in an effort to convey the results of what Watson is developed for (natural language analytics) to the medium in which he is tested, Jeopardy.

[Kaylia]

He has no capacity for knowing the others' answers. Watson is fed textual data, the answers are given to Trebec orally, there is no feasible way to give Watson knowledge of what the other players are doing.

Well, it would probably be quite "easy" (compared to the rest) to add voice recognition that can hear the player's answer (write down key word), analyse's Trebec's answer (just have to find one of the many possible negative/positive keyword) and compare it to the top answers found by the software.

If they didn't do it, it's probably because they didn't care.

[Julian]

I agree with Magus.

There are so many things you guys aren't considering. What if the person who gave the wrong answer had a formatting issue? It's not as simple as taking audio input and converting it to text. What if the person forgot to say "What is" or if he just said "leg" instead of "lost a leg"? Watson would have to interpret all of it, and determine why it was wrong and if it affects him.

Also, how often do you think Watson is going to come up with a wrong answer, not buzz in first, and have the other contestant give the exact same wrong answer?

[Tidane]

NOVA did an episode on Watson (http://www.pbs.org/wgbh/nova/tech/sm...-on-earth.html), which will answer some of the questions you have. For example, Watson was told the answers of his competitors, at some point. During NOVA, there is a point where he gets a category completely wrong, sees two answers by his competitors, and then is able to get the next answer himself. The problem here is that they are not real-time. Watson probably doesn't get the correct answer until after the question is answered correctly/by Trebek.

[ronin sparthos]

After watching Ken steamroll during his long winning streak, it was interesting watching him silently stand at that podium for the majority of the game.

The potential of AI capability is frightening really.

[Stiker]

Gotta love some people in here...you have an amazing example of the power of computing and AI and all some people can focus on is that Watson repeated an answer. Just wow.

Also, take a look at those sparing matches, Watson started flat out retarded. What IBM has done is absolutely amazing.

[Qalbert]

Because it is a glaring flaw that could have been easily dealt with. It stands out in contrast to how well it did everything else. No one was ever downplaying Watson, merely saying that there was a huge oversight. When you have created something nearly flawless any problem sticks out like a sore thumb, especially one so basic. If you made a robot to run in a race, and it had the perfect stride, and flawless form, and a semblance of balance and so on and so forth but overlooked programming it to recognize the starter's pistol or something equally small but glaring. It doesn't make it any less impressive, it jsut stands out.

[Norellicus]

Because it is a glaring flaw that could have been easily dealt with. It stands out in contrast to how well it did everything else. No one was ever downplaying Watson, merely saying that there was a huge oversight. When you have created something nearly flawless any problem sticks out like a sore thumb, especially one so basic. If you made a robot to run in a race, and it had the perfect stride, and flawless form, and a semblance of balance and so on and so forth but overlooked programming it to recognize the starter's pistol or something equally small but glaring. It doesn't make it any less impressive, it jsut stands out.

You are severely underestimating the amount of human AND machine effort required for functional auditory processing. It's a task that is certainly still on the table for research teams like Watson's, but doesn't manifest a lot of usefulness without first having a core system that can adequately process the resulting output; an interactive terminal is pointless without a backend that can process the inputs it receives and turn them around into useful feedback for the participant.


With regards to Watson himself and his performance, I'd be curious to see some data on things like the precise timing of when Watson receives the clue, and how long from the time it appears on screen for the contestants until he has actually started processing it; it seems to me that if he's being transmitted the text file at the precise moment it appears on screen, he has a distinct advantage over the other players, as even a speed reader wouldn't be able to process all of the words in a question in the time it takes Watson to parse what he receives.

I'd also like to see some captured data on how fast the buttons are being pressed by all three players. It was clear in the first little bit of the show that Brad was getting agitated at Watson being able to buzz in first almost every time he was highly confident in his answer, but was it just dumb luck or could he really think of it faster and react before either of the humans? It'll be interesting to see if Brad and Ken play a little differently in Double Jeopardy, buzzing in first reflexively and then formulating their answer during their timer.

[Stiker]

I highly doubt it was an oversight, but more of a "this isn't worth it" sort of thing. Happens in programming all the time. Odds of that exact situation coming up were rare, so it got bypassed. The assumption was that 99% of the time, the answer that Watson had as the most correct wouldn't be both wrong and said by another contestant.

Watson was being fed questions via text files, so you would have to add yet another delivery system to not only receive/parse the answers, but then also create additional logic to handle the other answers, and then calculate if Watson should click his buzzer based on the new information. And adding that on could also affect the speed at which Watson could respond with a second answer.

It's a calculated risk, and honestly, it made Watson seem a lot more human/genuine by "screwing up". Any thoughts of "well, Watson is being fed answers" died right there.