# Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs

**Authors:** Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, Yongbin Li  
**arXiv:** [2305.03111](https://arxiv.org/abs/2305.03111)  
**Date:** 2023-05-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.03111v2/

## Key points

- Text-to-SQL parsing involves converting natural language instructions into executable SQL queries
- Models like Codex and ChatGPT have shown impressive results in text-to-SQL tasks
- Existing benchmarks such as Spider and WikiSQL focus on limited database schema and contents, creating a gap between research and real-world applications
- Bird is a comprehensive benchmark designed for large-scale databases in text-to-SQL tasks
- Bird consists of 12,751 pairs of text-to-SQL data and encompasses 95 databases spanning 37 professional domains, with a total size of 33.4 GB
- Bird highlights challenges related to handling dirty database contents, leveraging external knowledge, and optimizing SQL efficiency for massive databases
- Text-to-SQL models need to comprehend database values to tackle these challenges effectively
- State-of-the-art models like ChatGPT have an execution accuracy of only 40.08%, far from human performance level (92.96%)
- Efficiency analysis provides insights into generating efficient SQL queries for industries
- Bird will contribute to advancing real-world applications of text-to-SQL research

## Summary

The field of text-to-SQL parsing, which involves converting natural language instructions into executable SQL queries, has gained significant attention in recent years. Notably, models like Codex and ChatGPT have shown impressive results in this task. However, existing benchmarks such as Spider and WikiSQL primarily focus on database schema with limited rows of database contents, creating a gap between academic research and real-world applications. To address this gap, the authors introduce Bird, a comprehensive benchmark designed for large-scale databases in text-to-SQL tasks. Bird consists of 12,751 pairs of text-to-SQL data and encompasses 95 databases spanning 37 professional domains, with a total size of 33.4 GB. The emphasis on database values in Bird highlights new challenges related to handling dirty database contents, leveraging external knowledge to connect natural language questions with database contents, and optimizing SQL efficiency specifically for massive databases. The authors argue that text-to-SQL models need to go beyond semantic parsing and also feature comprehension of database values to tackle these challenges effectively. Experimental results demonstrate the significance of incorporating database values for generating accurate text-to-SQL queries for large databases. Despite the effectiveness of state-of-the-art models like ChatGPT, their execution accuracy remains at only 40.08%, far from the human performance level of 92.96%. This indicates that substantial challenges still exist in this domain. Additionally, the authors provide an efficiency analysis to offer insights into generating text-to efficient SQL queries that can benefit industries. They believe that Bird will contribute to advancing real world applications of text to SQL research. Interested researchers can access the leaderboard and source code for Bird at https://birdbenchgithubio/.