# SpanBERT: Improving Pre-training by Representing and Predicting Spans

**Authors:** Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, Omer Levy  
**arXiv:** [1907.10529](https://arxiv.org/abs/1907.10529)  
**Date:** 2019-07-24  
**Summary page:** https://www.summarizepaper.com/en/paper/1907.10529v1/

## Key points

- SpanBERT is a method developed by Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy to improve the accuracy and efficiency of text spans.
- It introduces two key innovations: masking contiguous random spans during pre-training and training span boundary representations to predict masked spans without relying on token-level information.
- SpanBERT consistently outperforms BERT and other baselines in tasks like question answering and coreference resolution.
- With equivalent training data and model size as BERT-large, a single SpanBERT model achieves impressive F1 scores of 94.6% on SQuAD 1.1 and 88.7% on SQuAD 2.0.
- SpanBERT sets new state-of-the-art performance in coreference resolution with an F1 score of 79.6% on the OntoNotes dataset and significant gains in relation extraction with a score of 70.8% on the TACRED benchmark.
- It demonstrates improvements across various tasks including GLUE benchmarks, showcasing its effectiveness in capturing complex linguistic structures within text data.

## Summary

SpanBERT: Enhancing Performance in Natural Language Processing Tasks

 is a groundbreaking method developed by Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy that aims to improve the  and  of text spans. Unlike its predecessor BERT, SpanBERT introduces two key innovations: masking contiguous random spans instead of individual tokens during pre-training and training span boundary representations to predict the entire content of the masked span without relying on token-level information within it. The results of SpanBERT's implementation are impressive. It consistently outperforms BERT and other baselines in various span selection tasks such as question answering and coreference resolution. Notably, with equivalent training data and model size as BERT-large, a single SpanBERT model achieves remarkable F1 scores of 94.6% on SQuAD 1.1 and 88.7% on SQuAD 2.0. Additionally, SpanBERT sets a new state-of-the-art performance in coreference resolution with an F1 score of 79.6% on the OntoNotes dataset and achieves significant gains in relation extraction with a score of 70.8% on the TACRED benchmark. Moreover, SpanBERT demonstrates improvements across various tasks including GLUE (General Language Understanding Evaluation) benchmarks. This comprehensive evaluation showcases the effectiveness and versatility of SpanBERT in capturing complex linguistic structures and relationships within text data. In summary, in enhancing performance across a range of natural language processing tasks,.