# SliceGPT: Compress Large Language Models by Deleting Rows and Columns

**Authors:** Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, James Hensman  
**arXiv:** [2401.15024](https://arxiv.org/abs/2401.15024)  
**Date:** 2024-01-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.15024v1/

## Key points

- Authors Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman address the need to alleviate compute and memory resource costs associated with large language models.
- The paper introduces SliceGPT, a post-training sparsification scheme that replaces weight matrices with smaller dense matrices to reduce network embedding dimensions.
- SliceGPT demonstrates effectiveness by removing up to 25% of model parameters while maintaining high zero-shot task performance levels for LLAMA2-70B, OPT 66B, and Phi-2 models.
- One key advantage of SliceGPT is its ability to enable sliced models to run on fewer GPUs and operate faster without additional code optimization.
- On consumer GPUs (24GB) and A100 GPUs (40GB), SliceGPT significantly reduces total compute for inference on various models.
- The authors introduce a new insight called "Slice Transformer" in transformer networks which enhances SliceGPT's effectiveness and could inspire future approaches for reducing memory and computation demands in pre-trained models.

## Summary

In their paper titled "SliceGPT: Compress Large Language Models by Deleting Rows and Columns," authors Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman address the growing need to alleviate the substantial compute and memory resource costs associated with large language models used in natural language processing. The paper spans over 22 pages with 8 figures and has been accepted at ICLR24. The code for implementing SliceGPT is available at https://github.com/microsoft/TransformerCompression. introduces a novel post-training sparsification scheme that replaces each weight matrix in a model with a smaller (dense) matrix, effectively reducing the embedding dimension of the network. This addresses the need for  as a solution to resource constraints faced by large language models. Existing techniques have limitations such as additional data structures and limited speedup capabilities on current hardware. Through extensive experimentation,  demonstrates its effectiveness by removing up to 25% of model parameters (including embeddings) for LLAMA2-70B, OPT 66B, and Phi-2 models while maintaining impressive zero-shot task performance levels of 99%, 99%, and 90% respectively compared to the dense model. One key advantage of  is its ability to enable sliced models to run on fewer GPUs and operate faster without requiring additional code optimization. For instance, on 24GB consumer GPUs, it reduces total compute for inference on LLAMA2-70B to just 64% of that required by the dense model; on 40GB A100 GPUs, this reduction is even more significant at 66%. Furthermore, the authors introduce a new insight called  in transformer networks which underpins  effectiveness. They believe that this insight could inspire future avenues for reducing memory and computation demands for pre-trained models.