# Evaluating Large Language Models Trained on Code

**Authors:** Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder  
**arXiv:** [2107.03374](https://arxiv.org/abs/2107.03374)  
**Date:** 2021-07-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2107.03374v1/

## Key points

- Codex is a GPT language model that has been fine-tuned on publicly available code from GitHub.
- The authors focus on Codex's capabilities in writing Python code.
- Codex powers GitHub Copilot, a distinct production version.
- The authors introduce a new evaluation set called HumanEval to measure functional correctness in synthesizing programs from docstrings.
- Codex outperforms other models like GPT-3 and GPT-J in solving problems in HumanEval.
- Repeated sampling from the model is an effective strategy for generating working solutions to challenging prompts.
- However, Codex has limitations with long chains of operations and binding operations to variables in docstrings.
- The paper discusses potential broader impacts of deploying powerful code generation technologies like Codex, addressing concerns related to safety, security, and economics.

## Summary

In their paper, the authors introduce Codex, a GPT language model that has been fine-tuned on publicly available code from GitHub. They specifically focus on studying Codex's capabilities in writing Python code. It is worth noting that a distinct production version of Codex powers GitHub Copilot. To evaluate the model's performance, the authors release a new evaluation set called HumanEval, which measures functional correctness in synthesizing programs from docstrings. The results show that Codex outperforms other models such as GPT-3 and GPT-J. Specifically, Codex solves 28.8% of the problems in HumanEval, while GPT-3 solves 0% and GPT-J solves 11.4%. The authors also discover that repeated sampling from the model is an unexpectedly effective strategy for generating working solutions to challenging prompts. By using this method with 100 samples per problem, they are able to solve 70.2% of the problems. However, careful investigation of Codex reveals certain limitations; for instance it struggles with docstrings that describe long chains of operations and with binding operations to variables. Finally, the paper discusses the potential broader impacts of deploying powerful code generation technologies like Codex and addresses concerns related to safety, security and economics that arise from such deployments. Overall, this refined summary provides more detailed information about Codex's performance compared to other models and highlights its limitations when dealing with specific types of code-related tasks. Additionally it emphasizes the importance of considering potential implications when deploying advanced code generation technologies like Codex.