# Efficient Training of Language Models to Fill in the Middle

**Authors:** Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, Mark Chen  
**arXiv:** [2207.14255](https://arxiv.org/abs/2207.14255)  
**Date:** 2022-07-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2207.14255v1/

## Key points

- Autoregressive language models are effective in filling missing text using the fill-in-the-middle (FIM) technique.
- Using FIM does not compromise the generative capabilities of autoregressive language models.
- Future autoregressive language models should be trained using FIM by default for practicality and efficiency.
- Best practices for training FIM models include exploring key hyperparameters and making best-performing infilling model available through an API.
- The study discusses techniques and results from pretraining and finetuning experiments, providing recommendations for future research directions.

## Summary

In this study, we demonstrate the effectiveness of autoregressive language models in filling missing text through a simple data transformation technique known as fill-in-the-middle (FIM). Despite its growing popularity, our research shows that using FIM does not compromise the original generative capabilities of these models. We propose that future autoregressive language models should be trained using FIM by default due to its practicality and efficiency. To establish best practices for training FIM models, we conducted experiments exploring key hyperparameters and have made our best-performing infilling model available through our API. Additionally, we discuss various techniques and results from pretraining and finetuning experiments and provide recommendations for future research directions. Our study contributes valuable insights into enhancing autoregressive language models through FIM training methodology.