# Accuracy is Not All You Need

**Authors:** Abhinav Dutta, Sanjeev Krishnan, Nipun Kwatra, Ramachandran Ramjee  
**arXiv:** [2407.09141](https://arxiv.org/abs/2407.09141)  
**Date:** 2024-07-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.09141v1/

## Key points

- Study conducted by Abhinav Dutta, Sanjeev Krishnan, Nipun Kwatra, and Ramachandran Ramjee focuses on compression of Large Language Models (LLMs) using techniques like quantization.
- Traditional method of evaluating compressed models is by comparing accuracy on benchmarks to baseline model.
- Flips phenomenon occurs where correct answers in baseline model become incorrect in compressed model and vice versa.
- Extensive analysis across multiple compression techniques, models, and datasets shows significant discrepancies in user experience despite similar accuracy levels.
- Compressed models perform considerably worse than baseline models in free-form generative tasks according to qualitative and quantitative evaluations using MT-Bench.
- Researchers propose evaluating compression techniques using distance metrics like KL-Divergence and flips in addition to accuracy measurements for a more comprehensive understanding of model performance.

## Summary

In the study "Accuracy is Not All You Need," conducted by Abhinav Dutta, Sanjeev Krishnan, Nipun Kwatra, and Ramachandran Ramjee, the focus is on the compression of Large Language Models (LLMs) using techniques like quantization. The traditional method of evaluating these compressed models is by comparing their accuracy on various benchmarks to that of the baseline model. If the accuracies are similar, it is generally assumed that there has been minimal degradation in quality. However, even when accuracy levels are comparable between baseline and compressed models, a phenomenon known as flips occurs. Flips refer to instances where correct answers in the baseline model become incorrect in the compressed model and vice versa. The researchers conducted an extensive analysis across multiple compression techniques, models, and datasets to explore how the behavior of compressed models differs from that of baseline models when presented to end-users. Despite similar accuracy levels, they found significant discrepancies in user experience. Additionally, qualitative and quantitative evaluations using MT-Bench revealed that compressed models performed considerably worse than baseline models in free-form generative tasks. To address these issues, the researchers propose evaluating compression techniques using distance metrics in addition to accuracy measurements. They introduce two such metrics: KL-Divergence and flips, which demonstrate a strong correlation. By incorporating these metrics into evaluation processes,a more comprehensive understanding of model performance can be achieved. Overall,the study highlights the importance of considering factors beyond just accuracy when assessing compressed LLMs.By taking into account user experience and employing appropriate distance metrics for evaluation,researchers can gain deeper insights into the true impact of compression techniques on model quality and performance.