# When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset

**Authors:** Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson, Daniel E. Ho  
**arXiv:** [2104.08671](https://arxiv.org/abs/2104.08671)  
**Date:** 2021-04-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2104.08671v3/

## Key points

- The authors investigate the effectiveness of domain pretraining in legal natural language processing (NLP) tasks.
- They introduce a new dataset called CaseHOLD consisting of over 53,000+ multiple choice questions that require identifying the relevant holding of a cited case.
- Domain pretraining with a custom legal vocabulary using a corpus of approximately 3.5M decisions across all courts in the U.S. yields the most substantial performance gains on CaseHOLD and consistent improvements on two other legal tasks.
- The gain in F1 score on CaseHOLD is 7.2%, representing a 12% improvement over BERT.
- The level of performance increase in legal tasks is directly related to the domain specificity of the task.
- Researchers should engage in resource-intensive pretraining when the task exhibits sufficient similarity to the pretraining corpus.
- This study has implications for researchers deciding when to engage in data and resource-intensive pretraining to contribute to advancements in legal NLP and reduce disparities in access to justice within the U.S. legal system.
- Access to code, benchmark datasets, and Legal-BERT models is provided by the authors for further exploration and development.

## Summary

In this study, the authors investigate the effectiveness of domain pretraining in legal natural language processing (NLP) tasks. To address the lack of substantial gains from domain pretraining in law due to the simplicity of existing legal NLP tasks, they introduce a new dataset called CaseHOLD consisting of over 53,000+ multiple choice questions that require identifying the relevant holding of a cited case. This dataset is both legally meaningful and challenging from an NLP perspective. The authors evaluate the performance gains on CaseHOLD and other existing legal NLP datasets using different pretraining approaches. They find that while a Transformer architecture pretrained on a general corpus (BERT) improves performance, domain pretraining with a custom legal vocabulary using a corpus of approximately 3.5M decisions across all courts in the U.S. yields the most substantial performance gains on CaseHOLD and consistent improvements on two other legal tasks. The gain in F1 score on CaseHOLD is 7.2%, representing a 12% improvement over BERT. Furthermore, they demonstrate that the level of performance increase in legal tasks is directly related to the domain specificity of the task and suggest that researchers should engage in resource-intensive pretraining when the task exhibits sufficient similarity to the pretraining corpus. The findings of this study have implications for researchers deciding when to engage in data and resource-intensive pretraining as it can contribute to advancements in legal NLP and help reduce disparities in access to justice within the U.S. legal system. Overall, this research highlights the uniqueness of legal language and provides insights into when domain-specific pretraining can be beneficial for improving performance on challenging legal NLP tasks. The authors also provide access to their code, benchmark datasets, and Legal-BERT models for further exploration and development in this field.