# Program Synthesis with Large Language Models

**Authors:** Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, Charles Sutton  
**arXiv:** [2108.07732](https://arxiv.org/abs/2108.07732)  
**Date:** 2021-08-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2108.07732v1/

## Key points

- Large language models for program synthesis in general-purpose programming languages
- Evaluation of models on two benchmarks: MBPP and MathQA-Python
- MBPP dataset: 974 programming tasks for entry-level programmers
- MathQA-Python dataset: 23,914 problems involving synthesizing code from complex text
- Performance scales log-linearly with model size on both datasets
- Largest models can synthesize solutions for 59.6% of MBPP problems using few-shot learning
- Fine-tuning improves performance by approximately 10 percentage points across most model sizes
- Largest fine-tuned model achieves an accuracy of 83.8% on MathQA-Python dataset
- Incorporating human feedback reduces error rates by half compared to initial model predictions
- Error analysis reveals areas where models struggle and challenging types of programs to generate
- Models struggle to predict program outputs given specific inputs
- Insights into limitations and potential applications of large language models for program synthesis

## Summary

This paper investigates the capabilities of large language models for program synthesis in general-purpose programming languages. The authors evaluate several models with varying parameter sizes on two new benchmarks, MBPP and MathQA-Python, using both few-shot learning and fine-tuning approaches. The benchmarks are designed to assess the models' ability to generate short Python programs based on natural language descriptions. The MBPP dataset consists of 974 programming tasks aimed at entry-level programmers, while the MathQA-Python dataset contains 23,914 problems that involve synthesizing code from more complex text. The results show that the performance of program synthesis scales log-linearly with model size on both datasets. Even without finetuning on a code dataset, the largest models can successfully synthesize solutions for 59.6% of the MBPP problems using few-shot learning with well-designed prompts. Fine-tuning on a held-out portion of the dataset improves performance by approximately 10 percentage points across most model sizes. On the MathQA-Python dataset, the largest fine-tuned model achieves an accuracy of 83.8%. Furthermore, the authors explore how these models can engage in dialog about code by incorporating human feedback to enhance their solutions. They find that natural language feedback from humans reduces error rates by half compared to initial model predictions. Additionally, an error analysis reveals areas where these models struggle and identifies which types of programs are particularly challenging to generate. Finally, the authors investigate the semantic grounding of these models by fine-tuning them to predict program execution results; however even their best performing models generally struggle to predict program outputs given specific inputs. Overall, this study provides insights into the limitations and potential applications of large language models for program synthesis in general purpose programming languages.