LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

AI-generated keywords: LLM.int8() Int8 Matrix Multiplication Outlier Features NeurIPS2022

AI-generated Key Points

  • Large language models have revolutionized natural language processing but require significant GPU memory for inference
  • LLM.int8() is a procedure developed by researchers to address this issue
  • LLM.int8() cuts the memory needed for inference by half while retaining full precision performance
  • The researchers were able to load a 175B parameter checkpoint, convert it to Int8, and use it immediately without any performance degradation
  • LLM.int8() uses a two-part quantization procedure using vector-wise quantization with separate normalization constants for each inner product in the matrix multiplication to quantize most of the features
  • For emergent outliers, a new mixed-precision decomposition scheme isolates outlier feature dimensions into a 16-bit matrix multiplication while still more than 99.9% of values are multiplied in 8-bit
  • Outlier features were defined based on their magnitude compared to other features within the model and analyzed for their effect on model performance
  • LLM.int8() makes large language models more accessible by reducing GPU memory requirements during inference
Also access our AI generated: Comprehensive summary, Lay summary, Blog-like article; or ask questions about this paper to our AI assistant.

Authors: Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer

Extended NeurIPS2022 submission
License: CC BY 4.0

Abstract: Large language models have been widely adopted but require significant GPU memory for inference. We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers, which cut the memory needed for inference by half while retaining full precision performance. With our method, a 175B parameter 16/32-bit checkpoint can be loaded, converted to Int8, and used immediately without performance degradation. This is made possible by understanding and working around properties of highly systematic emergent features in transformer language models that dominate attention and transformer predictive performance. To cope with these features, we develop a two-part quantization procedure, LLM.int8(). We first use vector-wise quantization with separate normalization constants for each inner product in the matrix multiplication, to quantize most of the features. However, for the emergent outliers, we also include a new mixed-precision decomposition scheme, which isolates the outlier feature dimensions into a 16-bit matrix multiplication while still more than 99.9% of values are multiplied in 8-bit. Using LLM.int8(), we show empirically it is possible to perform inference in LLMs with up to 175B parameters without any performance degradation. This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs.

Submitted to arXiv on 15 Aug. 2022

Ask questions about this paper to our AI assistant

You can also chat with multiple papers at once here.

AI assistant instructions?

Results of the summarizing process for the arXiv paper: 2208.07339v1

The development of large language models has revolutionized natural language processing, but their inference requires significant GPU memory. To address this issue, a team of researchers has developed LLM.int8(), a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers. This method cuts the memory needed for inference by half while retaining full precision performance. The researchers were able to load a 175B parameter 16/32-bit checkpoint, convert it to Int8, and use it immediately without any performance degradation. The success of LLM.int8() is due to the researchers' understanding and working around properties of highly systematic emergent features in transformer language models that dominate attention and transformer predictive performance. They developed a two-part quantization procedure using vector-wise quantization with separate normalization constants for each inner product in the matrix multiplication to quantize most of the features. However, for the emergent outliers, they also included a new mixed-precision decomposition scheme that isolates the outlier feature dimensions into a 16-bit matrix multiplication while still more than 99.9% of values are multiplied in 8-bit. To find these outlier features, the researchers used an empirical approach and defined outliers based on their magnitude compared to other features within the model. By analyzing these outlier features, they were able to gain insight into how they affect model performance. Overall, LLM.int8() makes large language models much more accessible by reducing GPU memory requirements during inference which makes it possible to use OPT-175B/BLOOM on a single server with consumer GPUs without any performance degradation.
Created on 05 May. 2023

Assess the quality of the AI-generated content by voting

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Similar papers summarized with our AI tools

Navigate through even more similar papers through a

tree representation

Look for similar papers (in beta version)

By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.