# ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

**Authors:** Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning  
**arXiv:** [2003.10555](https://arxiv.org/abs/2003.10555)  
**Date:** 2020-03-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2003.10555v1/

## Key points

- Authors propose a new pre-training task called replaced token detection as an alternative to masked language modeling (MLM) methods like BERT
- Approach corrupts input tokens by replacing them with plausible alternatives from a small generator network
- Discriminative model is trained to predict whether each token in the corrupted input was replaced or not
- New pre-training task is more efficient than MLM because it is defined over all input tokens, not just the masked subset
- Contextual representations learned by this approach outperform those learned by BERT, given the same resources
- Particularly strong gains for small models, outperforming GPT on GLUE benchmark with less compute resources
- Performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute resources
- Effective and efficient alternative to MLM for improving contextual representations in natural language processing tasks

## Summary

The authors of the paper propose a new pre-training task called replaced token detection as an alternative to the commonly used masked language modeling (MLM) methods like BERT. Rather than masking input tokens with [MASK], the approach corrupts them by replacing some tokens with plausible alternatives sampled from a small generator network. Rather than training a model to predict the original identities of the corrupted tokens, a discriminative model is trained to predict whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate that this new pre-training task is more efficient than MLM because it is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by this approach substantially outperform those learned by BERT, given the same model size, data and compute resources. The gains are particularly strong for small models, as evidenced by a model trained on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Furthermore, this approach also works well at scale. It performs comparably to RoBERTa and XLNet while utilizing less than 1/4 of their compute resources and even outperforms them when using the same amount of compute. Overall, this research presents an effective and efficient alternative to MLM pre-training methods for improving contextual representations in natural language processing tasks.