# Knowledge Fusion of Large Language Models

**Authors:** Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, Shuming Shi  
**arXiv:** [2401.10491](https://arxiv.org/abs/2401.10491)  
**Date:** 2024-01-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.10491v2/

## Key points

- Authors explore merging existing pre-trained large language models (LLMs) to create a more powerful model
- Directly blending weights of LLMs with varying architectures is impractical
- Authors propose knowledge fusion for LLMs to transfer collective knowledge and unique strengths into a single model
- Implementing knowledge fusion on three popular LLMs - Llama-2, MPT, and OpenLLaMA - enhances performance in reasoning, commonsense understanding, and code generation
- Externalizing combined knowledge and strengths of source LLMs into a unified model elevates capabilities beyond individual source LLMs
- Code, model weights, and data are publicly available on GitHub for further exploration
- Research accepted for presentation at ICLR 2024 and contributes insights into improving large language models through knowledge fusion techniques

## Summary

In their paper titled "Knowledge Fusion of Large Language Models," authors Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi explore the concept of merging existing pre-trained large language models (LLMs) to create a more powerful model. This approach offers a cost-effective alternative to training LLMs from scratch and avoids redundant capabilities. However, directly blending the weights of LLMs with varying architectures is impractical. To address this challenge, the authors propose the idea of knowledge fusion for LLMs. By leveraging their generative distributions to transfer collective knowledge and unique strengths into a single model, this approach aims to combine the capabilities of different LLMs. Implementing knowledge fusion on three popular LLMs with different architectures - Llama-2, MPT, and OpenLLaMA - across various benchmarks and tasks, the authors demonstrate that fusing LLMs can enhance the performance of the target model in areas such as reasoning, commonsense understanding, and code generation. The findings suggest that by externalizing the combined knowledge and strengths of source LLMs into a unified model <KD>, it is possible to elevate its capabilities beyond those of any individual source LLM</KD>. The authors have made their code, model weights, and data publicly available on GitHub for further exploration. This research has been accepted for presentation at ICLR 2024 <KD>and contributes valuable insights into improving large language models through knowledge fusion techniques</KD>.