# Adding Instructions during Pretraining: Effective Way of Controlling Toxicity in Language Models

**Authors:** Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro  
**arXiv:** [2302.07388](https://arxiv.org/abs/2302.07388)  
**Date:** 2023-02-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.07388v1/

## Key points

- Two innovative strategies proposed to reduce toxicity of pretrained large language models (LMs):
- MEDA: Adding raw toxicity scores as meta-data to pretraining samples
- INST: Adding instructions in natural language form to indicate toxicity of samples
- PerspectiveAPI used to obtain toxicity scores for each sample
- LMs process documents into samples of 2000 tokens before obtaining PerspectiveAPI scores
- 15 datasets used for experiments, contributing to a corpus consisting of 339 billion tokens
- Results show that INST strategy reduces toxicity probability by up to 61% while maintaining accuracy on five benchmark NLP tasks and improving AUC scores on four bias detection tasks by 1.3%
- Techniques demonstrated generalizability by scaling number of training samples and model parameters

## Summary

In this study, the researchers propose two innovative strategies to reduce the toxicity of pretrained large language models (LMs) without compromising their utility. The first strategy, called MEDA, involves adding raw toxicity scores as meta-data to the pretraining samples. The second strategy, called INST, adds instructions in natural language form to indicate the toxicity of the samples. To obtain toxicity scores for each sample, the researchers use the widely accepted Commercial PerspectiveAPI. Since LMs have a maximum sequence length that is smaller than the size limit of PerspectiveAPI, they process documents in their dataset into samples of 2000 tokens before obtaining PerspectiveAPI scores. The researchers use a total of 15 datasets for their experiments including Common Crawl, The Pile, Books3, OpenWebText2, Stack Exchange, PubMed Abstracts, Wikipedia Gutenberg (PG-19), BookCorpus2 NIH ExPorter ArXiv GitHub and Pile-CC datasets which contribute to a corpus consisting of 339 billion tokens. The results show that INST strategy performs best in reducing toxicity probability by up to 61% while maintaining accuracy on five benchmark NLP tasks and improving AUC scores on four bias detection tasks by 1.3%. Additionally they demonstrate generalizability of their techniques by scaling number of training samples and model parameters. Overall this study presents an effective approach for controlling toxicity in LMs during pretraining by providing them with explicit instructions and incorporating toxicity scores as meta-data.