# BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

**Authors:** Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, Luke Zettlemoyer  
**arXiv:** [1910.13461](https://arxiv.org/abs/1910.13461)  
**Date:** 2019-10-29  
**Summary page:** https://www.summarizepaper.com/en/paper/1910.13461v1/

## Key points

- BART is a denoising autoencoder for pretraining sequence-to-sequence models
- It follows a two-step process: corrupting text and learning to reconstruct it
- BART utilizes a Transformer-based neural machine translation architecture
- Randomly shuffling sentences and using mask tokens yield the best performance for noising approaches
- BART excels in text generation tasks and performs well in comprehension tasks
- It achieves comparable results to RoBERTa on GLUE and SQuAD datasets, showcasing state-of-the-art performance in various tasks
- BART outperforms back-translation systems for machine translation by providing a 1.1 BLEU increase with only target language pretraining
- Ablation experiments within the BART framework identify factors that influence end-task performance
- Overall, BART proves to be an effective denoising autoencoder for pretraining sequence-to-sequence models, showing versatility across natural language processing tasks.

## Summary

BART is a denoising autoencoder designed for pretraining sequence-to-sequence models. It follows a two-step process: corrupting text with a noising function and then learning to reconstruct the original text. BART utilizes a Transformer-based neural machine translation architecture, which is an extension of BERT and GPT. The model explores various noising approaches and finds that randomly shuffling the order of sentences and using an in-filling scheme with mask tokens yield the best performance. When fine-tuned, BART excels in text generation tasks but also performs well in comprehension tasks. It achieves comparable results to RoBERTa on GLUE and SQuAD datasets, demonstrating state-of-the-art performance in abstractive dialogue, question answering, and summarization tasks with up to 6 ROUGE gains. Additionally, BART outperforms back-translation systems for machine translation by providing a 1.1 BLEU increase with only target language pretraining. The authors conduct ablation experiments within the BART framework to identify the factors that most influence end-task performance. Overall, BART proves to be an effective denoising autoencoder for pretraining sequence-to-sequence models, showcasing its versatility across various natural language processing tasks.