Study exposes major flaw in classic artificial intelligence test

computer
Credit: Public Domain

A serious problem in the Turing test for computer intelligence is exposed in a study published in the Journal of Experimental and Theoretical Artificial Intelligence.

If a machine were to 'take the Fifth Amendment' – that is, exercise the right to remain silent throughout the test – it could, potentially, pass the test and thus be regarded as a thinking entity, authors Kevin Warwick and Huma Shah of Coventry University argue. However, if this is the case, any silent entity could pass the test, even if it were clearly incapable of thought.

The test, devised in 1950 by pioneering computer scientist Alan Turing, assesses a machine's ability to exhibit intelligent behaviour indistinguishable from that of a human. Also known as the 'imitation game', it requires a human judge to converse with two hidden entities, a human and a machine, and then determine which is which.

Warwick and Shah's study looks at transcripts of a number of conversations from actual Turing tests in which the hidden machine remained silent. In each case, the human judge was unable to say for certain whether they were interacting with a person or a machine.

Thus, a machine could potentially pass the Turing test simply by remaining silent. The judge would be unable to determine whether the silent entity was a human choosing not to answer the questions, a smart machine that had decided not to reply, or a machine experiencing technical problems that prevented it from answering (as was actually the case in the transcripts studied).

Kevin Warwick said: "This begs the question, what exactly does it mean to pass the Turing test? Turing introduced his imitation game as a replacement for the question 'Can think?' and the end conclusion of this is that if an entity passes the test then we have to regard it as a thinking entity."

"However, if an entity can pass the test by remaining silent, this cannot be seen as an indication it is a thinking entity, otherwise objects such as stones or rocks, which clearly do not think, could pass the test. Therefore, we must conclude that 'taking the Fifth' fleshes out a serious flaw in the Turing ."


Explore further

Professor proposes alternative to 'Turing Test'

More information: Kevin Warwick et al. Taking the fifth amendment in Turing's imitation game, Journal of Experimental & Theoretical Artificial Intelligence (2016). DOI: 10.1080/0952813X.2015.1132273
Provided by Taylor & Francis Group
Citation: Study exposes major flaw in classic artificial intelligence test (2016, July 5) retrieved 17 September 2019 from https://techxplore.com/news/2016-07-exposes-major-flaw-classic-artificial.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.
1409 shares

Feedback to editors

User comments

Jul 05, 2016
" and the end conclusion of this is that if an entity passes the test then we have to regard it as a thinking entity.""

No it isn't. That's always been a non-sequitur conclusion of the Turing test, which patently does not test intelligence or the ability to think - of machines or otherwise - but the limits of people to recognize thought in others.

It is a test of our ability to test thinking systems - it attempts to answer whether we can reliably recognize other intelligence should we come across any, or even our own, by probing the limits where people fail to make a difference between a non-thinking machine and a person.

A sufficiently complex non-intelligent and non-thinking machine can always pass the Turing test by simply exhausting the tester.

Jul 05, 2016
The actual conclusion and important lesson of the Turing test is, that without a proper definition for intelligence or thought, we indeed cannot tell the difference.

So choosing to remain silent does not reveal a flaw in the Turing Test - on the contrary - it points the same way.

The issue of people regarding the Turing test as a definitive proof of intelligence and thought is simply because of behaviourist arguments from researchers working on AI. Since they don't have any proof that what they have made is intelligent, they fall back to the behaviourist argument of the duck - or, "It walks like a duck, talks like a duck, it is a duck".

But that simply depends on your ability to exhaustively test that it actually does walk and talk -exactly- like a duck in all circumstances, which you cannot.

Jul 05, 2016
" and the end conclusion of this is that if an entity passes the test then we have to regard it as a thinking entity."

Not if it doesn't participate in the test by remaining silent. If it remains silent then there is no way of knowing it was even being tested.
And this Tech splore section doesn't even seem to be allowing me to vote for Eikka's comments... (Ergo it failed this test)

Jul 05, 2016
"Not if it doesn't participate in the test by remaining silent."


It could be programmed to say "Hello" at the beginning and then shut up. The point of the article is that if the machine for example would crash half-way to the test, the person at the other end cannot tell whether its silence is due to it choosing not to answer.

That said, the person could not tell whether a human at the other end had a stroke and fell silent or simpy chose not to answer either - but that is simply the consequence of the Turing Test not testing for intelligence but for behaviour.


Jul 05, 2016
"If a machine were to 'take the Fifth Amendment'"
Well in this case it would be refusing to take the test. Every teacher will tell you what this means :)

Jul 05, 2016
I'm not sure I follow the logic of the article. If the subject doesn't answer any questions then the test outcome is simply unknown (as the test was effectively not taken). Not answering doesn't automatically equate to a 'pass'.
It's like trying to assess a student that refuses to answer math problems. You might flunk him, but it doesn't tell you anýthing about his mathematical abilities.

The problem is in forcing the judge into a pass/not pass situation. This judgement is supposed to be on the basis of information (i.e. an informed choice). Without any reply there is no information so the judgement - if it is forced - is no better than a random choice.

One could force an answer. Even if it always replies with a rote "I don't understand what you mean" one can make a judgement.
One could then pose a question that is so simple that anyone who can formulate "I don't understand what you mean" would be able to reply to it.

Jul 05, 2016
"a question that is so simple that anyone who can formulate "I don't understand what you mean" would be able to reply to it."

But they don't necessarily have to give a correct answer, because the test is not about whether you can answer simple questions, but whether you behave like a person.

A human could give you a nonsensical answer just to mess with you, so again you can't know from the result whether it was some huckster, or a computer malfunction. A person can choose to answer any or all of your questions with nonsense just as how they can choose not to answer at all.

Jul 05, 2016
In fact, an easy way to test whether a computer is a computer is to throw it a nonsensical question, or simply gibberish, and seeing if it attempts to answer it.

Jul 05, 2016
"a question that is so simple that anyone who can formulate "I don't understand what you mean" would be able to reply to it."

What do you mean? :-)

Jul 05, 2016
Too much lawyer logic going on here. It's simple. If it doesn't answer when spoken to the test is simply not a valid test and the machine fails.

Jul 09, 2016
"If it doesn't answer when spoken to the test is simply not a valid test and the machine fails."


And if a human person does that? Do you fail them too?

The Turing test is about whether a machine can pretend to be human. If a human may remain silent, then the machine may remain silent. Even though the result is inconclusive, you can't actually "fail" the test by not answering because then you'd have to fail the human for behaving like a human.

Jul 09, 2016
The unintended insight, that comes from a second look at the old analogy.

https://www.faceb...71396242

Jul 09, 2016
"If it doesn't answer when spoken to the test is simply not a valid test and the machine fails."


And if a human person does that? Do you fail them too?

The Turing test is about whether a machine can pretend to be human. If a human may remain silent, then the machine may remain silent. Even though the result is inconclusive, you can't actually "fail" the test by not answering because then you'd have to fail the human for behaving like a human.


Yes I would because the whole point of the test is to talk to the person trying to talk to them. I would also fire the person and replace them with someone who could follow simple directions.

Jul 09, 2016
This article and most of the replies, are together, the absolute dumbest article and set of replies, that I have ever read in any scientific publication.

All you have to do is add a statement to the Turing test stating that no reply, is not an acceptable reply, and if the computer fails to reply for some chosen time, then the test is terminated, just as it would be if there were a hardware problem resulting in a computer failure.

Why this even needs to be said, is beyond my comprehension, as it is as patently obvious, as patently obvious could ever be. And why this nonsense story is published here, is likely proof positive that there are real people that are actually no more conscious than computers. And why I bothered taking the time to comment here about all this, is probably a wake up call for me to consider committing myself to an insane asylum, because I am obviously losing my mind.

Jul 10, 2016
Turing was an English man, who would have regarded the fifth amendment as an anachronism. Silence should be seen as a refusal and therefore a failed or non conclusive test.

Jul 10, 2016
Time to close this 'research' institute and send Kevin and Hue home.
This kind of childish reasoning is sufficient for a 5 year old (I guess) and an huge insult to one of the most clever people in history.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more