# What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?

**Authors:** Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, Colin Raffel  
**arXiv:** [2204.05832](https://arxiv.org/abs/2204.05832)  
**Date:** 2022-04-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2204.05832v1/

## Key points

- Large pretrained Transformer language models can perform tasks they were not explicitly trained on (zero-shot generalization)
- Limited systematic comparison of different model architectures and pretraining objectives
- Comprehensive evaluation conducted on text-to-text models with three different architectures: causal decoder-only, non-causal decoder-only, and encoder-decoder
- Models trained using autoregressive language modeling and masked language modeling as pretraining objectives
- Evaluation done with and without multitask prompted finetuning
- Models trained with over 5 billion parameters for more than 170 billion tokens
- Causal decoder-only models trained on autoregressive language modeling objective show strongest zero-shot generalization after unsupervised pretraining
- Non-causal visibility models trained on masked language modeling objective followed by multitask finetuning perform best overall
- Pretrained non-causal decoder models can be adapted into generative causal decoder models using autoregressive language modeling as a downstream task
- Pretrained causal decoder models can be efficiently adapted into non-causal decoder models after multitask finetuning
- Insights provided for selecting appropriate architecture and pretraining objectives to maximize zero-shot generalization in large pretrained Transformer language models

## Summary

In this study, the authors investigate the performance of large pretrained Transformer language models in zero-shot generalization, where models can perform tasks they were not explicitly trained on. They highlight that different state-of-the-art models use varying architectures and pretraining objectives, but there has been limited systematic comparison of these factors. To address this gap, the authors conduct a comprehensive evaluation of modeling choices and their impact on zero-shot generalization. The focus is specifically on text-to-text models, and the authors experiment with three different model architectures: causal decoder-only, non-causal decoder-only, and encoder-decoder. These models are trained using two pretraining objectives: autoregressive language modeling and masked language modeling. Additionally, the models are evaluated both with and without multitask prompted finetuning. To ensure robust conclusions that can be applied to larger scales, the authors train models with over 5 billion parameters for more than 170 billion tokens. The experimental results reveal that causal decoder-only models trained on an autoregressive language modeling objective demonstrate the strongest zero-shot generalization after unsupervised pretraining. However, among all experiments conducted it is observed that models with non-causal visibility on their input trained using a masked language modeling objective followed by multitask finetuning perform best. This finding suggests considering adaptation of pretrained models across different architectures and objectives. Furthermore, the study reveals that pretrained non-causal decoder models can be adapted into generative causal decoder models by utilizing autoregressive language modeling as a downstream task. Similarly, pretrained causal decoder models can be efficiently adapted into non-causal decoder models to achieve competitive performance after multitask finetuning. Overall, this research provides valuable insights into selecting appropriate architecture and pretraining objectives for maximizing zero-shot generalization in large pretrained Transformer language models. The code and checkpoints used in this study are made available for further exploration.