# LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset

**Authors:** Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric. P Xing, Joseph E. Gonzalez, Ion Stoica, Hao Zhang  
**arXiv:** [2309.11998](https://arxiv.org/abs/2309.11998)  
**Date:** 2023-09-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.11998v1/

## Key points

- Authors emphasize the importance of studying interactions with large language models (LLMs) in real-world scenarios
- Introduce LMSYS-Chat-1M dataset consisting of one million real conversations involving 25 state-of-the-art LLMs
- Dataset collected from 210K unique IP addresses through Vicuna demo and Chatbot Arena website
- Overview of dataset's content, curation process, and basic statistics provided
- Demonstrates versatility through four use cases: content moderation models, safety benchmark, instruction following models, challenging benchmark questions
- Dataset serves as a valuable resource for researchers and practitioners interested in advancing LLM capabilities

## Summary

In their paper titled "LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset," authors Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric. P Xing, Joseph E. Gonzalez, Ion Stoica and Hao Zhang emphasize the increasing importance of studying how people interact with large language models (LLMs) in real-world scenarios due to their widespread use in various applications. To address this need for research and understanding of LLMs interactions with users in real world settings the authors introduce LMSYS-Chat-1M—a comprehensive dataset consisting of one million real conversations involving 25 state-of-the art LLMs. The dataset was collected from 210K unique IP addresses through the Vicuna demo and Chatbot Arena website. The authors provide an overview of the dataset's content and discuss its curation process including basic statistics such as topic distribution and diversity as well as originality and scale. They also demonstrate the versatility of the dataset through four use cases: developing content moderation models that perform similarly to GPT4; building a safety benchmark for evaluating LLMs; training instruction following models that perform similarly to Vicuna; and creating challenging benchmark questions to assess LLM capabilities. The authors believe that this dataset will serve as a valuable resource for researchers and practitioners interested in understanding and advancing LLM capabilities. The dataset is publicly available at [https://huggingface.co/datasets/lmsys/lmsys-chat-1m](https://huggingface.co/datasets/lmsys/lmsys-chat-1m). This expanded summary provides a more detailed description of the paper's content and highlights its significance in advancing research on large language models' interactions with users in real world settings.