# A Conversational Paradigm for Program Synthesis

**Authors:** Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, Caiming Xiong  
**arXiv:** [2203.13474](https://arxiv.org/abs/2203.13474)  
**Date:** 2022-03-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2203.13474v1/

## Key points

- Authors propose a conversational program synthesis approach using large language models
- Approach treats program synthesis as a multi-turn conversation between user and system
- Program synthesis formulated as a sequence prediction problem with natural language specification and conditional sampling of desired program
- Model called CodeGen trained on large language models for both natural language and programming language data
- Conversational capabilities emerge from autoregressive language modeling with weak supervision and scaling up data size and model size
- Multi-turn programming benchmark (MTPB) developed to evaluate model's behavior on conversational program synthesis
- Paradigm effectively addresses challenges of searching over vast program space and capturing user intent accurately
- CodeGen outperforms OpenAI's Codex on HumanEval benchmark when trained with up to 16B parameters on TPU-v4
- Authors plan to release JaxFormer, which includes checkpoints from their training library, as open source

## Summary

In their paper titled "A Conversational Paradigm for Program Synthesis," authors Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong propose a novel approach to program synthesis using large language models. Program synthesis involves generating a computer program as a solution to a given problem specification. The authors' conversational program synthesis approach aims to overcome the challenges faced by previous methods in searching through a vast program space and accurately capturing user intent. The proposed approach treats the process of writing a specification and program as a multi-turn conversation between a user and a system. It formulates program synthesis as a sequence prediction problem where the specification is expressed in natural language, and the desired program is conditionally sampled. To train their model called CodeGen, the authors use large language models trained on both natural language and programming language data. By leveraging weak supervision in the data and scaling up data size and model size, the authors observe that conversational capabilities emerge from simple autoregressive language modeling. To evaluate their model's behavior on conversational program synthesis, they develop a multi-turn programming benchmark (MTPB). Solving each problem in this benchmark requires multi-step synthesis through multiple turns of conversation between the user and the model. The findings of their study demonstrate the effectiveness of this paradigm in addressing challenges related to searching over a vast program space and accurately capturing user intent. Additionally, they show that CodeGen outperforms OpenAI's Codex on the HumanEval benchmark when trained with up to 16B parameters on TPU-v4. The authors plan to release JaxFormer which includes checkpoints from their training library as open source. Overall, this paper presents an innovative approach to program synthesis that leverages large language models and conversational interactions between users and systems. The results highlight the effectiveness of this paradigm in addressing challenges related to searching over a vast program space and accurately capturing user intent.