# GLM: General Language Model Pretraining with Autoregressive Blank Infilling

**Authors:** Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, Jie Tang  
**arXiv:** [2103.10360](https://arxiv.org/abs/2103.10360)  
**Date:** 2021-03-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2103.10360v2/

## Key points

- Authors propose General Language Model (GLM) based on autoregressive blank infilling
- GLM enhances blank filling pretraining with 2D positional encodings and arbitrary order predictions for spans
- GLM outperforms BERT, T5, and GPT in NLU, conditional generation, and unconditional generation tasks
- Even with 1.25 times the parameters of BERT Large, GLM achieves superior results from a single pretrained model
- Comparative results show GLMRoBERTa outperforms other models like BERTSumAbs and UniLMv2Base in abstractive summarization tasks
- Multi-task pretraining variations within the GLM framework - including GLMDoc and GLMSent - showcase competitive performance against established baselines

## Summary

, , , , 

In their paper titled "GLM: General Language Model Pretraining with Autoregressive Blank Infilling," authors Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, and Zhili propose a novel approach to pretraining language models that addresses the limitations of existing frameworks. They introduce the General Language Model (GLM) based on autoregressive blank infilling, which enhances blank filling pretraining by incorporating 2D positional encodings and enabling predictions in an arbitrary order for spans. This innovation leads to improved performance on NLU tasks compared to BERT and T5. GLM's flexibility allows it to be pretrained for different task types by adjusting the number and lengths of blanks. Across a diverse set of tasks spanning NLU, conditional generation, and unconditional generation, GLM surpasses BERT, T5, and GPT in performance while utilizing similar model sizes and data. Notably, even with 1.25 times the parameters of BERT Large, GLM achieves superior results from a single pretrained model. The authors also present comparative results on abstractive summarization using datasets such as CNN/DailyMail and XSum. They demonstrate that GLMRoBERTa outperforms other state-of-the-art models like BERTSumAbs (Liu and Lapata), UniLMv2Base (Bao et al. in terms of ROUGE scores on both datasets. The study further explores multi-task pretraining variations within the GLM framework - including GLMDoc and GLMSent - showcasing their competitive performance against established baselines. In conclusion, through innovative enhancements in blank infilling pretraining techniques coupled with multi-task learning capabilities, GLM emerges as a versatile language model that excels across diverse downstream tasks while demonstrating scalability with increased parameter sizes for optimal performance.