# Evaluating Quantized Large Language Models

**Authors:** Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, Yu Wang  
**arXiv:** [2402.18158](https://arxiv.org/abs/2402.18158)  
**Date:** 2024-02-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.18158v2/

## Key points

- Post-training quantization (PTQ) is a promising technique in large language models (LLMs) for reducing costs and enhancing efficiency.
- PTQ mitigates memory consumption and reduces computational overhead, offering significant benefits across diverse scenarios.
- Comprehensive evaluation of quantized LLMs is crucial for optimal performance and guiding selection of quantization methods.
- The study "Evaluating Quantized Large Language Models" by Shiyao Li et al. evaluates the impact of PTQ on Weight, Activation, and KV Cache across 11 model families ranging from 125M to 180B parameters.
- The evaluation covers various tasks including basic NLP tasks, emergent ability trustworthiness dialogue, and long-context tasks.
- State-of-the-art (SOTA) quantization methods are evaluated to demonstrate their applicability in real-world scenarios.
- Techniques like , , , , and are systematically evaluated for their effects on different aspects of LLMs.
- The authors provide recommendations for effectively applying quantization techniques and suggest future research directions in this field.
- The code used in the evaluation can be accessed at https://github.com/thu-nics/qllm-eval for further exploration or replication of the study's findings.

## Summary

In the realm of large language models (LLMs), post-training quantization (PTQ) has emerged as a promising technique to reduce costs and enhance efficiency. By effectively mitigating memory consumption and reducing computational overhead, PTQ offers significant benefits for LLMs across diverse scenarios. To ensure optimal performance, a comprehensive evaluation of quantized LLMs is crucial for guiding the selection of quantization methods. In this study titled "Evaluating Quantized Large Language Models," conducted by Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang, the authors delve into a thorough evaluation of the impact of PTQ on Weight, Activation, and KV Cache across 11 model families. These model families include OPT, LLaMA2, Falcon, Bloomz, Mistral, ChatGLM Vicuna LongChat StableLM Gemma Mamba with parameters ranging from 125M to 180B. The evaluation encompasses a wide range of tasks including basic NLP tasks emergent ability trustworthiness dialogue and long-context tasks. Additionally the study evaluates state-of-the-art (SOTA) quantization methods to showcase their applicability in real-world scenarios. Through extensive experiments and analysis presented in this paper , , , , and  techniques are systematically evaluated to summarize their effects on various aspects of LLMs. The authors provide valuable recommendations for applying quantization techniques effectively while also highlighting potential future directions for research in this field. For those interested in exploring further or replicating the study's findings, the code used in this evaluation can be accessed at https://github.com/thu-nics/qllm-eval. This comprehensive evaluation serves as a valuable resource for researchers, practitioners, and developers looking to optimize large language models through post-training quantization techniques.