# A Simple and Effective Pruning Approach for Large Language Models

**Authors:** Mingjie Sun, Zhuang Liu, Anna Bair, J. Zico Kolter  
**arXiv:** [2306.11695](https://arxiv.org/abs/2306.11695)  
**Date:** 2023-06-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.11695v2/

## Key points

- Authors: Mingjie Sun, Zhuang Liu, Anna Bair, J. Zico Kolter
- Introduce novel pruning method called Wanda (Pruning by Weights and activations) for Large Language Models (LLMs)
- Wanda prunes weights with smallest magnitudes multiplied by corresponding input activations on a per-output basis
- Does not require retraining or weight updates, making it practical for immediate use on pruned LLMs
- Motivation behind Wanda is emergent large magnitude features in LLMs
- Outperforms established baseline methods like magnitude pruning and competes competitively with recent approaches involving intensive weight updates
- Code available at https://github.com/locuslab/wanda
- More information on project page at https://eric-mingjie.github.io/wanda/home.html

## Summary

In their paper titled "A Simple and Effective Pruning Approach for Large Language Models," authors Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter introduce a novel pruning method called Wanda (Pruning by Weights and activations) designed to induce sparsity in pretrained Large Language Models (LLMs). As the size of LLMs increases, network pruning methods become essential to drop a subset of network weights while maintaining performance. Existing methods often require retraining or solving weight reconstruction problems, which can be computationally expensive for billion-scale LLMs. Wanda stands out as a straightforward yet effective approach that prunes weights with the smallest magnitudes multiplied by corresponding input activations on a per-output basis. Notably, Wanda does not necessitate retraining or weight updates, making it practical for immediate use on pruned LLMs. The motivation behind Wanda stems from the observation of emergent large magnitude features in LLMs. The authors conducted a comprehensive evaluation of Wanda on LLaMA and LLaMA-2 across various language benchmarks. Their results demonstrate that Wanda significantly outperforms established baseline methods such as magnitude pruning and competes competitively with recent approaches involving intensive weight updates. For those interested in implementing Wanda, the authors have made their code available at https://github.com/locuslab/wanda. Additionally, more information about the project can be found on the project page at https://eric-mingjie.github.io/wanda/home.html. This research provides valuable insights into efficient pruning techniques for optimizing the performance of Large Language Models without the need for extensive retraining or computational overhead.