# MiniPLM: Knowledge Distillation for Pre-Training Language Models

**Authors:** Yuxian Gu, Hao Zhou, Fandong Meng, Jie Zhou, Minlie Huang  
**arXiv:** [2410.17215](https://arxiv.org/abs/2410.17215)  
**Date:** 2024-10-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.17215v1/

## Key points

- Knowledge distillation (KD) is a valuable technique for training small, high-performing student language models using larger teacher LMs.
- Challenges in applying KD during the pre-training phase include efficiency, flexibility, and overall effectiveness.
- Existing methods struggle with high computational costs, tokenization matching issues, and loss of complexity and diversity in training data.
- MiniPLM is a novel approach that enhances LM pre-training by refining the distribution of training data with insights from the teacher's knowledge.
- MiniPLM conducts the process offline to enable efficient KD for multiple student LMs without significant training-time costs.
- MiniPLM operates solely on the training corpus, allowing for KD across different model families and enhancing flexibility.
- MiniPLM enriches the difficulty and diversity of training data to help student LMs acquire versatile knowledge for improved performance on downstream tasks.
- Extensive experiments have shown that MiniPLM boosts student LM performance, enhances language modeling capabilities, and reduces pre-training computation requirements.
- MiniPLM supports KD across model families and optimizes pre-training data utilization effectively.

## Summary

In the realm of natural language processing, knowledge distillation (KD) has proven to be a valuable technique for training small, high-performing student language models (LMs) using larger teacher LMs. While KD has shown effectiveness in fine-tuning models, challenges arise when applying it during the pre-training phase, particularly in terms of efficiency, flexibility, and overall effectiveness. Existing methods often struggle with high computational costs due to online teacher inference, require precise tokenization matching between teacher and student LMs, or risk losing the complexity and diversity present in the teacher-generated training data. To overcome these obstacles, a novel approach called MiniPLM has been introduced. This  aims to enhance LM pre-training by refining the distribution of training data with insights from the teacher's knowledge. One key feature of MiniPLM is its offline . By conducting this process offline, MiniPLM enables efficient KD for multiple student LMs without adding significant training-time costs. Additionally, MiniPLM operates solely on the training corpus, allowing for KD across different model families and enhancing overall flexibility. Moreover,the distinctions between large and small LMs to enrich the difficulty and diversity of the training data. This approach helps student LMs acquire versatile and sophisticated knowledge that can improve their performance on various downstream tasks. Extensive experiments have demonstrated that MiniPLM not only boosts student LM performance but also enhances language modeling capabilities while reducing pre-training computation requirements. Furthermore,to large pre-training scales as evidenced by extrapolation of scaling curves. The framework supports KD across model families and optimizes the utilization of pre-training data effectively. In conclusion,a significant advancement in knowledge distillation for pre-training language models. By incorporating Difference Sampling techniques to refine training distribution based on differences between large teacher LMs and small reference LMs, MiniPLM offers a comprehensive solution to challenges faced during pre-training stages. Its offline nature ensures efficiency while maintaining data complexity and diversity essential for robust model development.