# A Holistic Approach to Undesired Content Detection in the Real World

**Authors:** Todor Markov, Chong Zhang, Sandhini Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, Lilian Weng  
**arXiv:** [2208.03274](https://arxiv.org/abs/2208.03274)  
**Date:** 2022-08-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2208.03274v2/

## Key points

- Concerns about bias and fairness in models due to social biases present in training data
- Exploration of data augmentation techniques for lexicon robustness and model generalizability
- Improving multilingual support by evaluating performance on non-English text and potential adjustments to tokenization or model architecture
- Prioritizing red-teaming at scale to identify unknown failure cases efficiently, implementing a pipeline for model red-teaming
- Further experimentation with active learning strategy, including diversity sampling and selecting high-value data for labeling
- Developing methods to control outputs of large generative language models through building content detection models
- Acknowledgment of data workers' contributions and feedback received on the work
- Importance of meticulous attention to detail in data collection, labeling, model training, and active learning configurations for successful deployment of a practical moderation system

## Summary

In our work on building a robust natural language classification system for content moderation, we have identified several areas for future work and acknowledged limitations. One key area of concern is bias and fairness in our models, as they may exhibit biases towards certain demographic attributes due to social biases present in training data. We have attempted mitigation methods but acknowledge the need for ongoing research to improve model fairness. Additionally, we plan to explore data augmentation techniques to enhance lexicon robustness and model generalizability, especially as real-world data distributions evolve. Furthermore, we aim to improve multilingual support by evaluating performance on non-English text and potentially making adjustments to tokenization or model architecture. Red-teaming at scale is also a priority, as we seek more efficient ways to identify unknown failure cases for the model. We plan to implement a pipeline for model red-teaming inspired by existing interfaces to enhance efficiency. Our current active learning strategy will undergo further experimentation, including exploring diversity sampling and comparing different strategies for selecting high-value data for labeling. As large generative language models become more prevalent, it becomes crucial to develop methods of controlling their outputs. Our work aims to demonstrate one approach through building content detection models, with an eye towards refining our approach and aligning generative model outputs in the future. Acknowledgments are extended to data workers who have contributed significantly to this work, handling sensitive content and aiding in the development of automated systems for content moderation. We also express gratitude to individuals who provided feedback on this work. In conclusion, successful deployment of a practical moderation system requires meticulous attention to detail in data collection, labeling, model training, and active learning configurations. Our findings underscore the importance of detailed instructions and quality control throughout the process.