# AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models

**Authors:** Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W. Mahoney, Yaoqing Yang  
**arXiv:** [2410.10912](https://arxiv.org/abs/2410.10912)  
**Date:** 2024-10-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.10912v1/

## Key points

- Authors: Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W. Mahoney, Yaoqing Yang
- Title: "AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models"
- Objective: Enhance efficiency of large language models (LLMs) through pruning without performance loss
- Introduces AlphaPruning as a theoretically grounded method for layerwise sparsity ratios based on shape metrics
- Utilizes Heavy-Tailed Self-Regularization (HT-SR) Theory and empirical spectral densities (ESDs) to design improved pruning ratios
- Identifies variability in training levels and prunability across different layers in LLMs
- Achieves significant sparsity levels with reasonable perplexity scores; e.g., 80% sparsity in LLaMA-7B model
- Offers valuable insights and tools for enhancing efficiency and scalability of LLMs through advanced pruning techniques
- Code openly available on GitHub for reproducibility and further research

## Summary

In their paper titled "AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models," authors Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W. Mahoney, and Yaoqing Yang delve into the realm of pruning large language models (LLMs) to enhance model efficiency without sacrificing performance. The study builds upon recent advancements in LLM pruning techniques and introduces AlphaPruning as a more theoretically grounded method for allocating layerwise sparsity ratios based on shape metrics. This novel approach offers a more nuanced and effective way to optimize pruning across various layers of an LLM. The authors highlight a key limitation in existing LLM pruning strategies and address this challenge by leveraging Heavy-Tailed Self-Regularization (HT-SR) Theory and analyzing the empirical spectral densities (ESDs) of weight matrices to design improved layerwise pruning ratios for LLMs. Their analysis uncovers substantial variability in the training levels and prunability of different layers within an LLM. The experimental results presented by the authors demonstrate the efficacy of AlphaPruning in achieving significant sparsity levels while maintaining reasonable perplexity scores. Notably, AlphaPruning achieves an impressive 80% sparsity level in the LLaMA-7B model, marking a notable milestone in LLM pruning research. Furthermore, the authors have made their code openly available on GitHub, facilitating reproducibility and further exploration by the research community. Overall, this study contributes valuable insights and practical tools for enhancing the efficiency and scalability of large language models through advanced pruning techniques based on rigorous theoretical foundations.