# LLaMA Pro: Progressive LLaMA with Block Expansion

**Authors:** Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ying Shan, Ping Luo  
**arXiv:** [2401.02415](https://arxiv.org/abs/2401.02415)  
**Date:** 2024-01-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.02415v2/

## Key points

- Authors: Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ying Shan, Ping Luo
- Challenges faced by Large Language Models (LLMs) in acquiring new skills without compromising existing ones
- Introduction of a novel post-pretraining method involving expanding Transformer blocks and fine-tuning with new corpus data
- Approach enhances model's knowledge without catastrophic forgetting
- Development of LLaMA Pro-8.3B from experiments on code and math corpora
- LLaMA Pro-8.3B initialized from LLaMA2-7B demonstrates exceptional performance across general tasks, programming, and mathematics domains
- Introduction of LLaMA Pro-Instruct as an instruction-following counterpart to showcase advanced capabilities
- Benchmark evaluations show that LLaMA Pro outperforms existing models within the LLaMA family
- Model's potential for reasoning and addressing diverse tasks as an intelligent agent
- Integration of natural language processing with programming languages lays foundation for advancing language agents effective in various environments
- Study accepted by ACL 2024 Main Conference provides insights into progressive development of Large Language Models and their impact on enhancing language understanding and task performance across different domains

## Summary

In their paper titled "LLaMA Pro: Progressive LLaMA with Block Expansion," authors Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ying Shan, and Ping Luo explore the challenges faced by Large Language Models (LLMs) in acquiring new skills without compromising existing ones. They introduce a novel post-pretraining method for LLMs that involves expanding Transformer blocks and fine-tuning them using new corpus data. This approach effectively enhances the model's knowledge without experiencing catastrophic forgetting. The authors conduct experiments on code and math corpora, resulting in the development of LLaMA Pro-8.3B. This versatile foundation model is initialized from LLaMA2-7B and demonstrates exceptional performance across general tasks, programming, and mathematics domains. Additionally, they introduce LLaMA Pro-Instruct as an instruction-following counterpart to further showcase the model's advanced capabilities. Through comprehensive benchmark evaluations, the authors illustrate that LLaMA Pro outperforms existing models within the LLaMA family. The results highlight the model's potential for reasoning and addressing diverse tasks as an intelligent agent. By integrating natural language processing with programming languages,this research lays a solid foundation for advancing language agents that can operate effectively in various environments.Accepted by ACL 2024 Main Conference,this study provides valuable insights into the progressive development of Large Language Models and underscores their potential impact on enhancing language understanding and task performance across different domains.