# A Survey on Knowledge Distillation of Large Language Models

**Authors:** Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, Tianyi Zhou  
**arXiv:** [2402.13116](https://arxiv.org/abs/2402.13116)  
**Date:** 2024-02-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.13116v1/

## Key points

- Knowledge distillation (KD) techniques play a crucial role in transferring advanced capabilities from proprietary Large Language Models (LLMs) like GPT-4 to open-source models such as LLaMA and Mistral.
- KD acts as a vital conduit for infusing open-source LLMs with sophisticated functionalities and nuanced understandings characteristic of proprietary counterparts.
- The survey is structured around three foundational pillars - algorithm, skill, and verticalization - providing a thorough analysis of KD mechanisms, enhancement of cognitive abilities, and practical implications across various fields.
- Data augmentation (DA) emerges as a powerful paradigm within the KD framework to significantly enhance LLMs' performance by generating context-rich, skill-specific training data.
- By bridging the gap between proprietary and open-source LLMs, the survey aims to empower open-source models with advanced capabilities previously limited to proprietary giants, fostering a more inclusive and equitable landscape in AI advancements.

## Summary

This comprehensive survey delves into the intricate realm of knowledge distillation (KD) techniques applied to Large Language Models (LLMs), shedding light on the crucial role KD plays in transferring advanced capabilities from proprietary behemoths like GPT-4 to more accessible open-source models such as LLaMA and Mistral. In the ever-evolving landscape of artificial intelligence, this study meticulously examines the disparities between proprietary and open-source LLMs, showcasing how KD acts as a vital conduit for infusing the latter with the sophisticated functionalities and nuanced understandings characteristic of their proprietary counterparts. Structured around three foundational pillars - algorithm, skill, and verticalization - this survey provides a thorough analysis of KD mechanisms, the enhancement of specific cognitive abilities, and their practical implications across various fields. Notably, it explores the intricate interplay between data augmentation (DA) and KD, highlighting DA's emergence as a powerful paradigm within the KD framework to enhance LLMs' performance significantly. By leveraging DA to generate context-rich, skill-specific training data, KD transcends traditional boundaries, enabling open-source models to approximate the contextual adeptness, ethical alignment, and deep semantic insights typically associated with proprietary models. The ultimate goal of this work is to serve as an insightful guide for researchers and practitioners by offering a detailed overview of current methodologies in knowledge distillation while also proposing future research directions. By bridging the gap between proprietary and open-source LLMs, this survey underscores the potential for more accessible, efficient, and sustainable AI solutions. It aims to foster a more inclusive and equitable landscape in AI advancements by empowering open-source models with advanced capabilities previously limited to proprietary giants. The associated Github repository provides additional resources for those interested in exploring further developments in knowledge distillation of LLMs.