# Text Summarization with Pretrained Encoders

**Authors:** Yang Liu, Mirella Lapata  
**arXiv:** [1908.08345](https://arxiv.org/abs/1908.08345)  
**Date:** 2019-08-22  
**Summary page:** https://www.summarizepaper.com/en/paper/1908.08345v1/

## Key points

- The paper introduces the BERT model for text summarization
- A general framework is proposed for both extractive and abstractive models using BERT
- A novel document-level encoder based on BERT is introduced to capture document semantics
- Inter-sentence Transformer layers are stacked for extractive summarization
- Different optimizers are used for the encoder and decoder in abstractive summarization
- Two-staged fine-tuning approach improves summary quality
- Experimental results show state-of-the-art performance in both extractive and abstractive settings
- Code for the model is available at https://github.com/nlpyang/PreSumm

## Summary

The paper "Text Summarization with Pretrained Encoders" introduces the Bidirectional Encoder Representations from Transformers (BERT) model which has been used to improve various natural language processing tasks. The authors propose a general framework for both extractive and abstractive models that utilizes BERT. They introduce a novel document-level encoder based on BERT to capture the semantics of a document and obtain representations for its sentences. For extractive summarization, they stack inter-sentence Transformer layers on top of this encoder. For abstractive summarization, they propose a new fine-tuning schedule that uses different optimizers for the encoder and decoder to address the pretrained vs non-pretrained mismatch. The authors also demonstrate that a two-staged fine-tuning approach further improves summary quality. Experimental results on three datasets show that their model achieves state-of-the-art performance in both extractive and abstractive settings. The code for their model is available at https://github.com/nlpyang/PreSumm.