# FP4 All the Way: Fully Quantized Training of LLMs

**Authors:** Brian Chmiel, Maxim Fishman, Ron Banner, Daniel Soudry  
**arXiv:** [2505.19115](https://arxiv.org/abs/2505.19115)  
**Date:** 2025-05-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2505.19115v2/

## Key points

- Researchers have made significant progress in Large Language Models (LLMs) with models boasting hundreds of billions of parameters.
- Training and inference processes for LLMs require substantial computational power and memory bandwidth.
- Innovations in numerical precision and memory-efficient architectures are essential to address the challenges posed by the increasing size of LLMs.
- Fully quantized training (FQT) using 4-bit floating-point (FP4) precision has been developed for large-scale LLM training.
- The NVFP4 format has been identified as optimal for achieving superior results in FP4 training through key design choices like block sizes, scaling formats, and rounding methods.
- Stochastic rounding for backward and update passes and round-to-nearest for the forward pass enhance stability during training with FP4 precision.
- A 7-billion-parameter model was successfully trained on 256 Intel Gaudi2 accelerators using FP4 precision, demonstrating comparable downstream task performance to a standard BF16 baseline.
- This research showcases the feasibility and effectiveness of fully quantized training with FP4 precision, paving the way for future innovations in numerical precision optimization for large language models.

## Summary

In the rapidly evolving field of Large Language Models (LLMs), researchers have made significant strides in pushing the boundaries of natural language understanding and generation. With state-of-the-art models now boasting hundreds of billions of parameters, the capabilities of these models have expanded to new heights across various applications. However, this progress has come at a cost, as the training and inference processes require substantial computational power and memory bandwidth. To address the challenges posed by the increasing size of LLMs, innovations in numerical precision and memory-efficient architectures have become essential. While BF16 was once the predominant numerical format for pretraining LLMs due to its balance between precision and efficiency, recent advancements have prompted researchers to explore lower-precision alternatives to enhance computational efficiency and reduce memory requirements. One such groundbreaking development is fully quantized training (FQT) using predominantly 4-bit floating-point (FP4) precision for weights, activations, and gradients on datasets containing up to 200 billion tokens. Through extensive investigation into key design choices for FP4, including block sizes, scaling formats, and rounding methods, researchers have identified the NVFP4 format as optimal for achieving superior results. By employing stochastic rounding for backward and update passes and round-to-nearest for the forward pass, stability during training is enhanced. Moreover, a theoretical and empirical threshold has been established to determine when quantized training becomes less effective based on the relationship between gradient norm and quantization noise. Leveraging these insights, a 7-billion-parameter model was successfully trained on 256 Intel Gaudi2 accelerators using FP4 precision. The resulting FP4-trained model demonstrated downstream task performance comparable to a standard BF16 baseline, affirming that FP4 training is a practical and highly efficient approach for large-scale LLM training. In conclusion,this research represents a significant advancement in the field of LLMs by showcasing the feasibility and effectiveness of fully quantized training with FP4 precision. By providing a reference implementation for further exploration in this area (available at https://github.com/Anonymous1252022/fp4-all-the-way), this work paves the way for future innovations in numerical precision optimization for large language models.