# Counter Turing Test CT^2: AI-Generated Text Detection is Not as Easy as You May Think -- Introducing AI Detectability Index

**Authors:** Megha Chakraborty, S. M Towhidul Islam Tonmoy, S M Mehedi Zaman, Krish Sharma, Niyar R Barman, Chandan Gupta, Shreya Gautam, Tanay Kumar, Vinija Jain, Aman Chadha, Amit P. Sheth, Amitava Das  
**arXiv:** [2310.05030](https://arxiv.org/abs/2310.05030)  
**Date:** 2023-10-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.05030v1/

## Key points

- Concerns raised by the emergence of ChatGPT regarding risks and consequences associated with AI-generated artifacts
- US Copyright Office stance on registration of works lacking human authorship
- Drafting of regulations for AI development by both US and EU governments
- Introduction of Counter Turing Test (CT^2) benchmark to evaluate AGTD methods' robustness
- Proposal of AI Detectability Index (ADI) to rank Language Model Models (LLMs) based on detectability levels
- Importance of stylometric analysis in identifying unique traces left behind by authors in texts
- Challenges posed by high-entropy word replacements for watermark detection modules
- Unreliability of perplexity and burstiness metrics as indicators of human-written text, especially in academic writing or low-resource languages
- Emphasis on the need for robust AGTD methods and ethical considerations in AI development

## Summary

, , , , 

The emergence of ChatGPT has raised concerns about the risks and consequences associated with . The US Copyright Office has taken a stance on  for AI-generated artifacts, stating that works lacking human authorship will not be registered. Both the US and EU governments are in the process of drafting  for AI development. As generative AI garners attention, AI-generated text detection (AGTD) has become a focal point in research, with initial methods being proposed and subsequent techniques developed to bypass detection. To address the shortcomings of existing AGTD methods, a Counter Turing Test (CT^2) benchmark has been introduced to evaluate their robustness. Additionally, an AI Detectability Index (ADI) has been proposed to rank Language Model Models (LLMs) based on their detectability levels. Empirical findings suggest that larger LLMs tend to have higher ADI scores, indicating lower detectability compared to smaller models. Stylometric analysis plays a crucial role in identifying unique traces left behind by different authors in texts. Classifier-based approaches have been developed to identify instances generated by specific models but may struggle with new models or unfamiliar domains. <Ethical considerations surrounding AGTD methods highlight the potential misuse by bad actors for creating indistinguishable AI-generated fake news.</Ethical considerations>

Recent advancements in watermarking techniques have shown improvements in selecting keys and detecting watermarks. High-entropy word replacements pose challenges for watermark detection modules, making it difficult to identify newly generated text even after paraphrasing. Perplexity and burstiness metrics may not always be reliable indicators of human-written text, especially in academic writing or low-resource languages. Overall, these developments underscore the need for robust AGTD methods and  in AI development. The ongoing research aims to enhance detection capabilities while addressing potential misuse of AI-generated content.