# Towards Quantifying the Hessian Structure of Neural Networks

**Authors:** Zhaorui Dong, Yushun Zhang, Zhi-Quan Luo, Jianfeng Yao, Ruoyu Sun  
**arXiv:** [2505.02809](https://arxiv.org/abs/2505.02809)  
**Date:** 2025-05-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2505.02809v1/

## Key points

- Recent empirical studies show that the Hessian matrix of neural networks often exhibits a near-block-diagonal structure.
- Two primary forces shaping the Hessian structure within NNs were identified: a "static force" inherent in the network's architecture design and a "dynamic force" that emerges during training.
- The study conducted rigorous theoretical analysis of the "static force" at random initialization stages for linear models and 1-hidden-layer networks, focusing on behavior with respect to mean-square error (MSE) loss and Cross-Entropy (CE) loss for classification tasks.
- Comparison between limit distributions of diagonal and off-diagonal blocks within the Hessian matrix was made using insights from random matrix theory, showing that as the number of classes approaches infinity, the block-diagonal structure becomes more pronounced.
- Detailed numerical investigations were conducted to elucidate how both static and dynamic forces influence the near-block-diagonal nature of the Hessian matrix, particularly relevant for large language models operating with high values of class sizes ranging from 32k to 128k.
- The study extended its analysis to 1-hidden-layer networks, highlighting key sub-matrices related to hidden layers and output layers within the Hessian matrix.
- A systematic approach was proposed to effectively tackle challenges arising from dependencies related to activation functions like ReLU and loss functions like CE loss, providing valuable insights into optimizing large-scale models in real-world applications.

## Summary

Recent empirical studies have shown that the Hessian matrix of neural networks (NNs) often exhibits a near-block-diagonal structure. However, the theoretical underpinnings of this phenomenon remain elusive. To shed light on this aspect, a study was conducted to investigate the forces shaping the Hessian structure within NNs. Two primary forces were identified: a "static force" inherent in the network's architecture design and a "dynamic force" that emerges during training. The study delved into a rigorous theoretical analysis of the "static force" at random initialization stages for linear models and 1-hidden-layer networks, focusing on their behavior with respect to mean-square error (MSE) loss and Cross-Entropy (CE) loss for classification tasks. By leveraging insights from random matrix theory, comparisons were made between the limit distributions of diagonal and off-diagonal blocks within the Hessian matrix. Notably, as the number of classes (C) approaches infinity, the block-diagonal structure becomes more pronounced. This finding is particularly relevant for large language models (LLMs), such as Llama 2 and DeepSeek-V3, which operate with significantly high values of C ranging from 32k to 128k. Detailed numerical investigations were also conducted to elucidate how both static and dynamic forces influence the near-block-diagonal nature of the Hessian matrix. The study extended its analysis to 1-hidden-layer networks, highlighting key sub-matrices related to hidden layers and output layers within the Hessian matrix. Through meticulous theoretical derivations and technical contributions addressing dependencies arising from activation functions like ReLU and loss functions like CE loss, a systematic approach was proposed to effectively tackle these challenges. Overall, this comprehensive study provides valuable insights into the intricate dynamics governing Hessian structures in neural networks with potential implications for optimizing large-scale models operating with extensive class sizes in real-world applications.