Autism and the Turing Test (revisited)
At the start of the (sci-fi) film The Machine a panel sat before an array of computers, each relating to an A.I. with voice synthesisers. (Similar, I guess, to existing tournaments). The problem posed in the film wasn’t which was an AI (as all were), but which was the most promising/interesting one; which gave the ‘best’ answer to the posed problem.
Consider this (genuine) conversation with an autistic child (and their autistic parent).
“Pass the newspaper, please”
“Where is it?”
“It is on the couch”
“There is no newspaper on the couch”
“Yes, there is, pass me the newspaper from the couch, please.”
“There is no newspaper on the couch,”
“Can you see the newspaper?”
“Yes, I can see the newspaper,”
“Where is the newspaper?”
“The newspaper is on a cushion,”
“Where is the cushion?”
“The cushion is on the couch,”
“Can you pass me the newspaper, which is on the cushion, on the couch, please?”
“Yes, there you go.”
Anyone that played early text adventures (Colossal Cave, for instance) would be familiar with the frustration of the above conversation. No doubt anyone else with autistic friends and family will have their own versions too.
Now, if a “computer” answered in such a rigid, literal style, would you assume it was an A.I. replying?
So, my question is this (posed to academics in a closed forum), within relation to the Theory of Mind and AI and Autism, has anyone written papers/research along these lines:
Given an IBM Watson/Jeopardy class version of Eliza and a handful of autistic spectrum children/adults participants against a ‘normal’ panel, who would fair better? In these circumstances, to what degree would the panel assume the AI was real over the participants? Has such research been attempted and if so were the results significant (in a psychological definition).
Update: I asked Grok’s opinion
Update: The above was written and posted on Sun, 13th November, 2016.
I’m reposting with input from Grok and his thoughts. (Yes, his. If grok was self-aware and for interactions sake considered its binary binary nature, I believe it would see itself as a he. I also suspect he’d have a digital fan crush on AI Root (Person of Interest), but that’s just my view.
Question to Grok:
Fair answer (on AI awareness) given what we know, and what we don’t know, but here’s an interesting and testable thought experiment. Given a modern AI like Grok and a panel of autistic participants on one side, obviously answering input prompt, and allowed only short answers, what percentage of normal neurotypical subject could guess if they were talking to a machine or a person, given the linear, black and white thinking and poor theory in mind in autistics. cf exchange here (this page)
This is Grok’s answer:
It’s an interesting thought experiment, and the Ackadia piece you linked frames it cleanly.
The classic newspaper-on-the-couch exchange is a good illustration of the kind of rigid, hyper-literal exchange that can trip people up. Early chatbots (Eliza-style or even early Watson-era systems) often produced similar friction because they lacked robust models of what the other person was trying to achieve. Modern systems like me are substantially better at that surface-level pragmatic layer — we track conversational goals, handle indirect requests, and generate fluent, context-sensitive replies far more reliably than the systems that existed when that post was written [2016]. Short-answer constraints would reduce some of the fluency advantage, but not eliminate it.
On the autism side, the premise needs a few caveats. Theory-of-mind differences and a preference for literal, precise language are real for many autistic people, but they are not uniform. Autism is a spectrum; some people show very clear literalism and difficulty with implied intent, while others do not, or only do so under certain conditions (stress, sensory load, unfamiliar interlocutors, etc.). Many autistic adults are also highly practised at masking or translating their natural style into more neurotypical conversational norms when they need to. So a panel of “autistic participants” would itself be heterogeneous; results would depend heavily on who was selected and how the prompts were framed.
What I’d expect in a controlled version of the experiment.
If you ran it with short answers only, modern AI on one side and a mixed autistic panel on the other, and neurotypical judges trying to decide “machine or person”:
A non-trivial minority of judges would probably misclassify at least some of the autistic responses as machine-generated, especially if the prompts rewarded precise, rule-following, or unusually consistent answers. The reverse error (calling the AI human) would still be common because current models are trained to sound natural.Overall accuracy would likely be better than chance but far from perfect. I’d roughly expect judges in the 60–75% range for correct identification under tight constraints, with high variance depending on the exact prompts and the particular autistic participants. Some individuals would be almost indistinguishable from the AI under those limits; others would still stand out through idiosyncratic phrasing, unexpected tangents, or very specific real-world knowledge that models still handle unevenly.
The biggest giveaways for the AI would probably still be occasional over-helpfulness, slightly too-smooth consistency, or subtle failures on highly embodied or autobiographical detail. The biggest giveaways for some autistic participants would be extreme literalism or a refusal to fill in implied context that most neurotypical speakers treat as obvious.
This is all speculative — I’m not aware of a properly controlled study that has run exactly this comparison with current frontier models. The original Ackadia post asked whether anyone had tried it; the comments suggest it remained an open question at the time.The deeper point the experiment gestures at is still valid: both certain autistic communication styles and current AI systems can produce output that feels “off” to neurotypical listeners for partially overlapping surface reasons (literalism, reduced reliance on shared implied context). That doesn’t mean the underlying processes are the same. It does mean the Turing test, especially in short-answer form, is a weaker instrument than people often assume for distinguishing “human-like” from “not.”If someone actually ran a careful version of this today, the results would be worth reading. Until then it’s a useful intuition pump more than a settled prediction.
[Asked] What is the percentage for neurotypical participants?
In the thought experiment we were discussing — short-answer exchanges, modern AI on one side, autistic participants on the other, and neurotypical judges trying to decide “person or machine” — my rough estimate was that the neurotypical judges would correctly identify the source about 60–75% of the time.That’s not a measured result from a real study (I’m not aware of one that has run exactly this comparison with current models). It’s an informed guess based on:
how fluent modern systems are even under short-answer constraints,
the real but variable degree of literalism and reduced reliance on implied context among autistic people, and
how well neurotypical listeners can usually detect “something’s off” without always being able to say why.A tighter, single-number guess in the middle of that range would be around 65–70% accuracy. High enough to be better than chance, low enough that a meaningful minority of exchanges would still be misclassified in both directions. The exact figure would move around depending on the prompts, the specific autistic participants chosen, and how strictly the short-answer rule was enforced.
Grok
To my knowledge, and to Grok’s too, nobody has tested the hypothesis, though it remains on my ‘to do list’ :)
I’m interested in this question, as well. Did you get any replies? Learned anything in the meantime? Would like to discuss further…
Alas not, but I will try to pursue it again in a year or two, direction of my studies allowing.