# Scalable Data Annotation Pipeline for High-Quality Large Speech Datasets Development

**Authors:** Mingkuan Liu, Chi Zhang, Hua Xing, Chao Feng, Monchu Chen, Judith Bishop, Grace Ngapo  
**arXiv:** [2109.01164](https://arxiv.org/abs/2109.01164)  
**Date:** 2021-09-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2109.01164v1/

## Key points

- The paper introduces a human-in-the-loop (HITL) data annotation pipeline that combines the advantages of both humans and machines to generate high-quality, large-scale speech datasets.
- The pipeline uses machine pre-labeling and fully manual auditing to annotate datasets quickly, accurately, and cost-effectively.
- Quality control mechanisms such as blind testing, behavior monitoring, and data validation have been adopted in the annotation pipeline to mitigate potential bias introduced by machine-generated labels.
- A/B testing and pilot studies demonstrated the HITL pipeline can improve annotation speed and capacity by at least 80%, while maintaining comparable or higher quality than manual double pass annotation.
- The scalable pipeline is being used to create ultra-high volume off-the-shelf (UHV-OTS) speech corpora for multiple languages with the capability to expand to 10,000+ hours per language annually.
- Customized datasets can be produced from the UHV-OTS corpora using dynamic packaging.
- Appen will donate a number of free speech datasets from the UHV OTS each year under CC BY SA license to support academic and open source community research.
- The authors are releasing the code of the data preprocessing and pre-tagging pipeline under Apache 2.0 license to allow reproduction of their results.

## Summary

The paper "Scalable Data Annotation Pipeline for High-Quality Large Speech Datasets Development" introduces a human-in-the-loop (HITL) data annotation pipeline that combines the advantages of both humans and machines to generate high-quality, large-scale speech datasets. The pipeline uses machine pre-labeling and fully manual auditing to annotate datasets quickly, accurately, and cost-effectively. Quality control mechanisms such as blind testing, behavior monitoring, and data validation have been adopted in the annotation pipeline to mitigate potential bias introduced by machine-generated labels. The authors conducted A/B testing and pilot studies that demonstrated the HITL pipeline can improve annotation speed and capacity by at least 80%, while maintaining comparable or higher quality than manual double pass annotation. They are leveraging this scalable pipeline to create ultra-high volume off-the-shelf (UHV-OTS) speech corpora for multiple languages with the capability to expand to 10,000+ hours per language annually. Customized datasets can be produced from the UHV-OTS corpora using dynamic packaging. The UHV-OTS is a long term Appen project that aims to support commercial and academic research data needs in speech processing. To support academic and open source community research under the CC BY SA license, Appen will donate a number of free speech datasets from the UHV OTS each year. Additionally, the authors are releasing the code of the data preprocessing and pre tagging pipeline under Apache 2.0 license to allow reproduction of their results. Overall, this paper presents an innovative approach for generating high quality speech datasets at scale using a combination of human expertise and machine learning techniques. The HITL pipeline has significant implications for advancing research in speech processing by providing researchers with access to large volumes of annotated data that can be used for training machine learning models.