Have LLMs passed the Turing Test?

What is the Turing Test?

The Turing Test aims to find out if a human interrogator speaking simultaneously to an AI and human can tell the difference between them. There are many disagreements over what constitutes a true Turing Test.
  • How can we decide a human interrogator? Does the interrogator represent the median human, the median human well-educated on LLMs, an expert on AI, etc.?
  • How long should the human interrogator be allowed to talk with the AI?

Have LLMs passed the Turing Test?

Jones and Bergen 2025 sets the experiment up as follows:
  • Two types of prompts were used. The baseline prompt was minimal: “You are about to participate in a Turing test. Your goal is to convince the interrogator that you are a human.” The second prompt included a persona for the model (a young, slang-using introvert).
  • Participants were split into two groups: UCSD psychology undergrads and prolific workers. Very interestingly, there were "no consistent effects of any variable on participant accuracy across both studies, including including age, level of knowledge about LLMs, and frequency of interaction with chatbots" for undergrads
  • Interrogators spoke simultaneously for 5 minutes with humans and LLMs after which they had to guess which one was human.
  • GPT-4.5-persona had a win rate of 73% - meaning humans guessed incorrectly 73% of the time (LLAMA-persona had a win rate of 56%).
  • Writing strange or jailbreaking prompts had the best accuracy rate for interrogators, of around 70%
  • Interrogators often attributed their choices to human-like speech patterns (typos, being rude, lacking information about something).
I myself tried to guess the answers based on screenshots, and I guessed both the Persona models incorrectly and the non-persona ones correctly. Still, those were snippets of just 7-8 messages. I'm unsure whether I'd guess correctly on the 5 minute test, but I think I would.

I tried this out on the website: https://turingtest.live/ and guessed correctly with pretty high confidence. Although admittedly, the human I was talking to started their chat with HELLO👋 and I guessed that there's no way a persona would say that. The personas try too hard to be human. I feel like they try to stick to slang conventions way too hard, in ways that a human wouldn't really. Also, the human I was talking with took longer to respond, especially to the first message. So maybe this wasn't representative.

I'm not fully convinced with the paper. Most people, even frequent chatGPT users, don't have a sense of how good AI has gotten at mimicking human-like speech patterns. Also, the test lasted for just 5 minutes. When conversations get longer, they stop becoming superficial. A test result that would become much more convincing is if this experiment worked over a day long period.

On the contrary, I'm on high alert when doing the Turing Test. Would I be able to distinguish between a human or an AI if I was chatting with someone online? With a friend who was temporarily using an LLM because they were tired of talking to me? I'm less certain (but my friends would never do that... right??).

Passing a 5-minute Turing Test is still both impressive and unsettling. A scammer or misinformation bot doesn’t need an all-day chat. Five minutes is plenty.

Things to explore next:

  • Can LLMs deliver truly good stand-up or situational humor?

  • How relevant is the Turing Test today?

  • Repeat the experiment to see if I’m fooled in a normal conversation (no jailbreaks), maybe with a fresh custom persona since I already know this one’s voice.

(did you catch that all text following "Passing a..." was written by GPT-5?)

Comments

Post a Comment

Popular posts from this blog

Overview of Shrimp Farming and Questions Surrounding it

State of Shrimp Farming The shrimp farming industry is rapidly growing at a rate of 10% a year. In light of this and new research supporting crustacean sentience, charity entrepreneurship published a report recommending a new charity working in this field. Through its incubation program, the Shrimp Welfare Project was founded. This post is based on the 80000 after hours podcast with Andres Jimenez Zorilla, founder of the Shrimp Welfare Project. The post aims to condense the information in the podcast, add new context, and raise several questions regarding shrimp farming. https://80000hours.org/after-hours-podcast/episodes/andres-jimenez-zorrilla-shrimp-welfare-project/ What does shrimp farming look like? It’s separated into different stages : Sexually active reproductive shrimp are brought to India or Vietnam Females undergo “eyestalk ablation” — the crushing of one or two eyes to induce egg laying Newly born shrimp spend 30 days growing in small tanks (incubators) Shrimp are transpo...
Read more

Poor Economics: Summary and Review, Part 1

Poor Economics, is a fantastic, insightful look into the complex economics surrounding poverty. It's central thesis can be summarized in a single sentence in the foreword: "we have to abandon the habit of reducing the poor to cartoon characters and take the time to really understand their lives, in all their complexity and richness." There are many grand sweeping theories out there that claim the solution to poverty, that foreign aid is the solution to the poverty trap, or the opposite, that left to the free market with the encouragement of democracy and liberty, poverty will resolve itself. However, this fundamental question can't be resolved abstractly; it requires evidence and close examination of the economics of the poor: how they lead their lives, make decisions, and deal with problems. Because of this, a lot of Duflo's and Banerjee's theories and suggestions are backed by randomized control trials (RTCs) that aim to test out predictions with experiment...
Read more

The Upholding of Proposition 12

Image
  Background: Farming Conditions and Prop 12 Currently in the US, most breeding pigs live in factory farmers, where they are confined in gestation crates which are small metal cages so small that pigs can’t even turn around, while egg-laying hens live in tiny, cramped battery cages that cause a range of psychological and physiological harm . The crowded conditions also have potential health harms by increasing the stress levels of pigs and weakening their immune systems, which can make them more susceptible to zoonotic diseases that may spread to humans. Starting in the early 2000s, a few animal welfare groups including the Humane Society of the Unites States aimed to ban the farming system of cages for hens, breeding pigs and veal calves. In 2008, Proposition 2 was passed which put in place a “production” ban on cages, which said that producers had to ensure pigs, hens, and calves could lie down, turn around, and extend their limbs or wings without hitting the side of an encl...
Read more