# ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning

**Authors:** Petter Törnberg  
**arXiv:** [2304.06588](https://arxiv.org/abs/2304.06588)  
**Date:** 2023-04-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.06588v1/

## Key points

- The paper explores the accuracy, reliability, and bias of the Large Language Model (LLM) ChatGPT-4 in classifying political affiliation on Twitter.
- The study compares ChatGPT-4 to manual annotation by expert classifiers and crowd workers.
- ChatGPT-4 achieves higher accuracy, higher reliability, and equal or lower bias than human classifiers.
- LLMs will have a substantial impact on using textual data in social sciences by enabling interpretive research at scale.
- Traditional forms of interpretive textual research tend to miss patterns of language use that are not directly observable because they are realized across thousands or millions of words of running text and not categorical but probabilistic.

## Summary

This paper explores the accuracy, reliability, and bias of the Large Language Model (LLM) ChatGPT-4 in classifying the political affiliation of a Twitter poster based on tweet content. The study compares ChatGPT-4 to manual annotation by both expert classifiers and crowd workers, which are considered the gold standard for such tasks. The research uses Twitter messages from United States politicians during the 2020 election as a ground truth to measure accuracy. The findings reveal that ChatGPT-4 achieves higher accuracy, higher reliability, and equal or lower bias than human classifiers. The LLM is capable of correctly annotating messages that require reasoning based on contextual knowledge and inferences around author intentions - traditionally seen as uniquely human abilities. These findings suggest that LLMs will have a substantial impact on using textual data in social sciences by enabling interpretive research at scale. The study also highlights how texts are culturally and socially situated within ideas, values, and beliefs about the world within which humans operate. Their interpretation requires deep contextual knowledge and the ability to "put ourselves in the shoes of others" - seen as distinctly human resources and capacities. While interpretation lies at the heart of social sciences, it has been viewed as almost intrinsically qualitative with an interpretive or hermeneutic element. Researchers seeking to use textual data have had to pursue one of two strategies: expert coders or crowd-workers on online labor platforms like Amazon's MTurk. However, these strategies have limitations such as being slow, costly, biased, having limited attention-spans hence restricting interpretive research to relatively small-N studies leading to complaints of lacking rigor and replicability while limiting potential in Big Data research. Conventional forms of interpretive textual research tend to miss patterns of language use that are not directly observable because they are realized across thousands or millions of words of running text and not categorical but probabilistic. Computational methods for analyzing textual data such as Natural Language Processing and Machine Learning have evolved quickly in recent years.