# ShortGPT: Layers in Large Language Models are More Redundant Than You Expect

**Authors:** Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, Weipeng Chen  
**arXiv:** [2403.03853](https://arxiv.org/abs/2403.03853)  
**Date:** 2024-03-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.03853v1/

## Key points

- Large Language Models (LLMs) have significantly increased in size, with billions or trillions of parameters in current iterations.
- Many layers within LLMs exhibit similarities and some contribute minimally to the network's functionality.
- Researchers introduced a metric called Block Influence (BI) to assess individual layer importance in LLMs.
- The BI scores are used for a pruning technique called layer removal, leading to the development of ShortGPT.
- ShortGPT has surpassed previous state-of-the-art methods in model pruning, demonstrating its effectiveness in simplifying LLM architectures.
- ShortGPT is compatible with quantization-like methods, allowing for further reductions in parameters and computational complexity.
- The presence of substantial redundancy in existing model architectures highlights the potential for more efficient optimization and advancements in model compression techniques.

## Summary

In the realm of Large Language Models (LLMs), the pursuit of enhanced performance has led to a significant increase in model size. Current iterations boast billions or even trillions of parameters. However, a recent study has shed light on an intriguing discovery - many layers within LLMs exhibit striking similarities. Additionally, some layers contribute minimally to the overall functionality of the network. Building upon this revelation, researchers have introduced a novel metric known as Block Influence (BI) to assess the importance of individual layers within LLMs. This groundbreaking approach has paved the way for a simple yet effective pruning technique dubbed layer removal. By leveraging BI scores to identify and eliminate redundant layers within LLMs, researchers have unveiled a new method named ShortGPT. Experimental results have showcased that ShortGPT surpasses previous state-of-the-art (SOTA) methods in model pruning, underscoring its efficacy in streamlining LLM architectures. Moreover, what sets ShortGPT apart is its compatibility with quantization-like methods, enabling further reductions in both parameters and computational complexity. The ability to achieve superior results through straightforward layer removal highlights the presence of substantial redundancy in existing model architectures. This innovative research not only offers a more efficient approach to optimizing LLMs but also underscores the potential for continued advancements in model compression techniques.