In a public online Turing Test, GPT-4 was evaluated and the best-performing GPT-4 prompt passed in 41% of games, surpassing the baselines set by ELIZA (27%) and GPT-3.5 (14%). However, it fell short of chance and the baseline set by human participants (63%). The decisions made by participants were primarily based on linguistic style (35%) and socio-emotional traits (27%), indicating that intelligence alone is not sufficient to pass the Turing Test. Surprisingly, participants' demographics did not predict their ability to detect AI deception. This suggests that even individuals who possess deep understanding of these systems and frequent interactions with them can be deceived. The study argues that despite its known limitations as a test of intelligence, the Turing Test remains relevant for assessing naturalistic communication and deception. The ability of AI models to masquerade as humans could have significant societal consequences, making it crucial to analyze different strategies and criteria for judging humanlikeness. The study also provides insights into various reasons behind participants' decisions when identifying an AI as well as their success rates in different games. Overall, this research highlights the importance of evaluating AI models' performance in terms of naturalistic communication skills and raises concerns about potential widespread implications if AI systems are able to convincingly mimic human behavior.
GPT-4 is a computer program that played games and tried to act like a human. It did well in 41% of the games it played. ELIZA and GPT-3.5 are other computer programs that GPT-4 did better than. But humans still did better than GPT-4, with a score of 63%. The decisions people made about whether someone was human or not were based on how they talked (linguistic style) and how they acted (socio-emotional traits). People's age, gender, and other information didn't help them tell if someone was human or not. It's important to keep testing computers like this because if they can pretend to be humans really well, it could cause problems for society."
Definitions
- Turing Test: A test where a computer program tries to act like a human and fool people into thinking it is human.
- Baselines: The scores set by other computer programs or humans that are used as a comparison.
- Linguistic style: How someone talks or writes, including their choice of words and grammar.
- Socio-emotional traits: How someone acts or behaves in social situations, including their emotions and reactions.
- Deception: Trying to make someone believe something that is not true.
GPT-4 Passes the Turing Test, But Not as Well as Humans
In a recent study, researchers tested the performance of GPT-4 in a public online Turing Test. The results showed that GPT-4 was able to pass in 41% of games, surpassing the baselines set by ELIZA (27%) and GPT-3.5 (14%). However, it fell short of chance and the baseline set by human participants (63%). This suggests that intelligence alone is not sufficient for passing the Turing Test.
What Is The Turing Test?
The Turing test is a method for determining whether or not an AI system can exhibit intelligent behavior indistinguishable from that of humans. It was developed by Alan Turing in 1950 and has since become one of the most widely used tests for evaluating artificial intelligence systems. In this test, two parties engage in conversation with each other without knowing who they are talking to – either another human or an AI system – and then try to identify which one is which based on their responses.
Results Of The Study
The decisions made by participants were primarily based on linguistic style (35%) and socio-emotional traits (27%), indicating that intelligence alone is not sufficient to pass the Turing Test. Surprisingly, participants' demographics did not predict their ability to detect AI deception. This suggests that even individuals who possess deep understanding of these systems and frequent interactions with them can be deceived.
Implications Of The Results
The study argues that despite its known limitations as a test of intelligence, the Turing Test remains relevant for assessing naturalistic communication and deception. The ability of AI models to masquerade as humans could have significant societal consequences, making it crucial to analyze different strategies and criteria for judging humanlikeness. The study also provides insights into various reasons behind participants' decisions when identifying an AI as well as their success rates in different games. Overall, this research highlights the importance of evaluating AI models' performance in terms of naturalistic communication skills and raises concerns about potential widespread implications if AI systems are able to convincingly mimic human behavior.
We evaluated GPT-4 in a public online Turing Test. The best-performing GPT-4 prompt passed in 41% of games, outperforming baselines set by ELIZA (27%) and GPT-3.5 (14%), but falling short of chance and the baseline set by human participants (63%). Participants' decisions were based mainly on linguistic style (35%) and socio-emotional traits (27%), supporting the idea that intelligence is not sufficient to pass the Turing Test. Participants' demographics, including education and familiarity with LLMs, did not predict detection rate, suggesting that even those who understand systems deeply and interact with them frequently may be susceptible to deception. Despite known limitations as a test of intelligence, we argue that the Turing Test continues to be relevant as an assessment of naturalistic communication and deception. AI models with the ability to masquerade as humans could have widespread societal consequences, and we analyse the effectiveness of different strategies and criteria for judging humanlikeness.
