# CodeS: Towards Building Open-source Language Models for Text-to-SQL

**Authors:** Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, Hong Chen  
**arXiv:** [2402.16347](https://arxiv.org/abs/2402.16347)  
**Date:** 2024-02-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.16347v1/

## Key points

- Language models like ChatGPT and GPT-4 are commonly used for Text-to-SQL tasks but have drawbacks such as unclear model architectures, data privacy risks, and high inference overheads.
- CodeS is a series of pre-trained language models developed by researchers led by Haoyang Li et al., tailored specifically for text-to-SQL tasks.
- CodeS stands out for being fully open-source and achieving superior accuracy with smaller parameter sizes compared to existing large language models (LLMs).
- The development of CodeS involved addressing research challenges through incremental pre-training using a SQL-centric corpus, schema linking strategies, and bi-directional data augmentation techniques.
- CodeS was evaluated on various datasets including Spider, BIRD, Spider-DK, Spider-Syn, Spider-Realistic, Dr.Spider, and real-world financial and academic datasets.
- Experimental results showed that CodeS outperformed existing models in accuracy and robustness across multiple challenging text-to-SQL benchmarks.
- This research sets a new standard for open-source language models tailored for complex NLP tasks like text-to-SQL translation.

## Summary

, , , , 

In the field of natural language processing, language models have demonstrated impressive capabilities in translating natural language questions into SQL queries, a task known as Text-to-SQL. However, many of the current state-of-the-art approaches rely on closed-source large language models (LLMs) like ChatGPT and GPT-4. These models, while powerful, come with drawbacks such as unclear model architectures, data privacy risks, and high inference overheads. To overcome these limitations, a team of researchers led by Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li and Hong Chen have introduced CodeS - a series of pre-trained language models tailored specifically for the text-to-SQL task. What sets CodeS apart is its fully open-source nature and its ability to achieve superior accuracy despite having much smaller parameter sizes compared to existing LLMs. The development of CodeS involved addressing various research challenges. To enhance its SQL generation abilities, an incremental pre-training approach was adopted using a carefully curated SQL-centric corpus. Additionally, challenges related to schema linking and rapid domain adaptation were tackled through strategic prompt construction and a bi-directional data augmentation technique. The effectiveness of CodeS was evaluated extensively across multiple datasets including popular benchmarks like Spider and BIRD as well as robustness-diagnostic benchmarks such as Spider-DK, Spider-Syn, Spider-Realistic and Dr.Spider. Furthermore, real-world datasets designed for financial and academic applications were also included in the evaluation process. The experimental results showcased that CodeS outperformed existing models in terms of accuracy and robustness on almost all challenging text-to-SQL benchmarks. This groundbreaking work by the research team not only advances the field but also sets a new standard for open-source language models tailored specifically for complex NLP tasks like text-to-SQL translation. The paper detailing this research has been accepted for presentation at the SIGMOD 2024 conference.