# Knowledge Distillation of Large Language Models

**Authors:** Yuxian Gu, Li Dong, Furu Wei, Minlie Huang  
**arXiv:** [2306.08543](https://arxiv.org/abs/2306.08543)  
**Date:** 2023-06-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.08543v1/

## Key points

- Knowledge Distillation (KD) is used to reduce the computational demand of large language models (LLMs)
- Previous KD methods have focused on white-box classification models or training small models to imitate black-box model APIs
- Limited exploration on effectively distilling knowledge from white-box generative LLMs
- Authors propose MiniLLM, a method for distilling smaller language models from larger generative language models
- Forward Kullback-Leibler divergence (KLD) objective replaced with reverse KLD to prevent overestimation of low-probability regions
- Effective optimization approach developed to learn this objective
- MiniLLM demonstrated through extensive experiments in an instruction-following setting
- Results show more precise responses, higher overall quality, lower exposure bias, better calibration, and improved long-text generation performance
- Scalable and applicable to different model families ranging from 120M to 13B parameters
- Code and model checkpoints will be released for further exploration
- White-box KD becomes valuable for leveraging teacher parameters in addition to existing techniques like black-box KD
- MiniLLM offers a promising approach for reducing computational demands while maintaining high-quality responses

## Summary

Knowledge Distillation (KD) is a technique used to reduce the computational demand of large language models (LLMs). While previous KD methods have primarily focused on white-box classification models or training small models to imitate black-box model APIs like ChatGPT, there is still limited exploration on effectively distilling knowledge from white-box generative LLMs. This becomes increasingly important as LLMs continue to prosper. In this work, the authors propose MiniLLM, a method for distilling smaller language models from larger generative language models. To address the challenges specific to generative language models, the authors replace the forward Kullback-Leibler divergence (KLD) objective in standard KD approaches with reverse KLD. This helps prevent the student model from overestimating low-probability regions of the teacher distribution. The authors also develop an effective optimization approach to learn this objective. The effectiveness of MiniLLM is demonstrated through extensive experiments in an instruction-following setting. The results show that MiniLLM models generate more precise responses with higher overall quality, lower exposure bias, better calibration, and improved long-text generation performance. Importantly, MiniLLM is scalable and can be applied to different model families ranging from 120M to 13B parameters. The authors plan to release their code and model checkpoints for further exploration. In addition to existing techniques such as black-box KD which only rely on teacher predictions, white-box KD becomes increasingly valuable for both research communities and industry sectors due to its ability to leverage teacher parameters. By refining and expanding upon existing knowledge distillation methods for generative LLMs, MiniLLM offers a promising approach for reducing computational demands while maintaining high-quality responses. Overall, this work contributes to the advancement of knowledge distillation techniques specifically tailored for generative language models and provides insights into improving response precision, diversity, and overall performance in various applications.