# An Empirical Study of LLaMA3 Quantization: From LLMs to MLLMs

**Authors:** Wei Huang, Xingyu Zheng, Xudong Ma, Haotong Qin, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, Michele Magno  
**arXiv:** [2404.14047](https://arxiv.org/abs/2404.14047)  
**Date:** 2024-04-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.14047v2/

## Key points

- The <Organization>LLaMA</Organization> family is a key player in open-source <Organization>Large Language Models (LLMs)</Organization, foundational for <Organization>Multimodal Large Language Models (MLLMs)</Organization>.
- LLaMA3 models have shown remarkable performance due to extensive pre-training on a dataset with over 15 trillion tokens.
- The study explores the impact of low-bit quantization on LLaMA3 and future LLM iterations, focusing on mitigating performance degradation during compression.
- Ten post-training quantization methods and LoRA-finetuning approaches are evaluated across bit-widths from 1 to 8 bits using diverse datasets to assess adaptability under reduced bit-width scenarios.
- Performance of the <Organization>LLaVA-Next-8B</Organization> model, based on LLaMA3, with ultra-low bits shows noticeable degradation in linguistic and visual contexts, highlighting the need for improvement.
- The study aims to drive future model developments by enhancing accuracy at lower bit-width configurations for more practical applications in various domains.

## Summary

The <Organization>LLaMA</Organization> family has emerged as a leading force in the realm of open-source <Organization>Large Language Models (LLMs)</Organization> and serves as a foundational backbone for <Organization>Multimodal Large Language Models (MLLMs)</Organization>, which are extensively utilized in <Organization>Computer Vision (CV)</Organization> and <Organization>Natural Language Understanding (NLU)</Organization> tasks. The latest iteration, LLaMA3 models, have recently been introduced with remarkable performance achievements stemming from extensive pre-training on a vast dataset comprising over 15 trillion tokens. Recognizing the prevalent use of low-bit quantization techniques for LLMs in resource-constrained environments, this study delves into the potential of LLaMA3 when subjected to quantization at low bit-widths. This exploration aims to shed light on novel insights and challenges associated with the low-bit quantization process for LLaMA3 and upcoming LLM iterations, particularly focusing on mitigating performance degradation issues commonly encountered during LLM compression. The research meticulously evaluates ten existing post-training quantization methods along with LoRA-finetuning approaches applied to LLaMA3 across bit-widths ranging from 1 to 8 bits using diverse datasets. By scrutinizing the low-bit quantization performance of LLaMA3 through these methodologies, the study aims to uncover the model's adaptability under reduced bit-width scenarios. Furthermore, an assessment is conducted on the performance of the <Organization>LLaVA-Next-8B</Organization> model, based on LLaMA3, utilizing 2-4 ultra-low bits in conjunction with post-training quantization techniques. The experimental findings reveal that despite its capabilities, LLaMA3 exhibits noticeable degradation in both linguistic and visual contexts when subjected to ultra-low bit-width conditions. This underscores a significant performance gap that necessitates addressing in future advancements within this domain. It is anticipated that the empirical insights garnered from this study will play a pivotal role in propelling future model developments forward. By striving towards achieving higher accuracy levels at lower bit-width configurations, there is a concerted effort to enhance practicality within <Organization>Large Language Models (LLMs)</Organization> and <Organization>Multimodal Large Language Models (MLLMs)</Organization>, ultimately paving the way for more efficient and effective applications across various domains.