# Open-Source Large Language Models Outperform Crowd Workers and Approach ChatGPT in Text-Annotation Tasks

**Authors:** Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Juan Diego Bermeo, Maria Korobeynikova, Fabrizio Gilardi  
**arXiv:** [2307.02179](https://arxiv.org/abs/2307.02179)  
**Date:** 2023-07-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.02179v1/

## Key points

- Study examines performance of open-source Large Language Models (LLMs) in text annotation tasks
- Compares LLMs with proprietary models like ChatGPT and human-based services like MTurk
- Four distinct datasets used: tweets on content moderation, tweets from US Congress members, newspaper articles on content moderation, replication of first dataset
- Annotation tasks include relevance and topic detection
- ChatGPT achieves best performance in most tasks
- Open-source LLMs outperform MTurk and show competitive potential against ChatGPT in specific tasks
- Highlights cost-effectiveness, transparency, reproducibility, and superior data protection offered by open-source LLMs like HuggingChat and FLAN
- Study provides insights into performance of different language models in text annotation tasks
- Emphasizes potential of open-source LLMs as alternatives to proprietary models and human-based services

## Summary

This study examines the performance of open-source Large Language Models (LLMs) in text annotation tasks and compares it with proprietary models like ChatGPT and human-based services such as MTurk. The researchers used four distinct datasets, including tweets related to content moderation, tweets from members of the US Congress, newspaper articles on content moderation, and a replication of the first dataset. They implemented several annotation tasks, including relevance and topic detection. The findings show that while ChatGPT achieves the best performance in most tasks, open-source LLMs not only outperform MTurk but also demonstrate competitive potential against ChatGPT in specific tasks. This research highlights the cost-effectiveness, transparency, reproducibility and superior data protection offered by open-source LLMs like HuggingChat and FLAN. Overall, this study provides valuable insights into the performance of different language models in text annotation tasks and emphasizes the potential of open-source LLMs as alternatives to proprietary models and human-based services.