# FGMP: Fine-Grained Mixed-Precision Weight and Activation Quantization for Hardware-Accelerated LLM Inference

**Authors:** Coleman Hooper, Charbel Sakr, Ben Keller, Rangharajan Venkatesan, Kurt Keutzer, Sophia Shao, Brucek Khailany  
**arXiv:** [2504.14152](https://arxiv.org/abs/2504.14152)  
**Date:** 2025-04-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2504.14152v1/

## Key points

- Large Language Model (LLM) inference efficiency improvement through quantization
- Introduction of Fine-Grained Mixed Precision (FGMP) quantization as a hardware-software co-design methodology post-training
- Development of a policy using perturbation and Fisher information to determine precision levels for weight and activation blocks
- Proposal of sensitivity-weighted clipping approach for fine-grained quantization
- Hardware augmentations for maximizing efficiency benefits, including support at block granularity and mixed-precision activation quantization unit
- Achieved less than 1% perplexity degradation on Wikitext-103 with FGMP compared to all-FP8 baseline design
- Consumes 14% less energy during inference and requires 30% less weight memory
- Analysis includes examining percentage of blocks retained in high precision across different layers, optimizing performance beyond linear layers to fully connected layers like FC2

## Summary

, , , , 

In the realm of Large Language Model (LLM) inference efficiency improvement, quantization emerges as a potent tool by leveraging low-precision datapaths for enhanced energy efficiency and reduced memory footprint. However, accurately quantizing LLM weights and activations to lower precision without compromising model accuracy remains a challenge. To address this issue, we introduce Fine-Grained Mixed Precision (FGMP) quantization - a hardware-software co-design methodology post-training that maintains accuracy while quantizing the majority of weights and activations to reduced precision. Our work presents several key contributions
1) We have developed a policy that utilizes perturbation in each value, weighted by the Fisher information, to determine which weight and activation blocks should be retained in higher precision. This approach ensures accuracy preservation by identifying the specific blocks that need higher precision to minimize perturbation in model loss. 2) Additionally, we propose a sensitivity-weighted clipping approach for fine-grained quantization, aiding in retaining accuracy for blocks quantized to lower precision. 3) Furthermore, we introduce hardware augmentations aimed at maximizing the efficiency benefits of FGMP quantization. This includes datapath support for FGMP at block granularity and a mixed-precision activation quantization unit enabling real-time assignment of activation blocks to high or low precision with minimal runtime and energy overhead. Our hardware implementation utilizes NVFP4 as the low-precision datatype and FP8 as the high-precision datatype. Through efficient FGMP quantization, our design achieves less than 1% perplexity degradation on Wikitext-103 for the Llama-2-7B model compared to an all-FP8 baseline design while consuming 14% less energy during inference and requiring 30% less weight memory. Moreover, our analysis delves into examining the percentage of blocks retained in high precision across different layers, providing valuable insights into optimizing performance. This comprehensive approach extends beyond linear layers to encompass fully connected layers such as FC2, enhancing overall model efficiency and accuracy.