# Data Predictability Shapes Weibull Weight-Scale Growth in Transformer Training

**Authors:** Tiexin Ding  
**arXiv:** [2608.23573](https://arxiv.org/abs/2608.23573)  
**Date:** 2026-06-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2608.23573v1/

## Key points

- Two-parameter Weibull distribution effectively summarizes weight magnitudes
- Scale parameter λ crucial for capturing training-induced changes in weight magnitudes
- Learning-rate-conditioned law governs how scale parameter λ grows: λ^2 - λ0^2 = C0(η) + C1(η)(Hr - D)^0.59
- Forward predictive model successfully recovers held-out within-family weight growth with 5.7% relative error
- Importance of understanding classical metrics and estimator saturation in training dynamics
- Correlations along the corruption axis do not imply causal mediation, but indicate predictive relations between pre-training data statistics and post-training weight-scale outcomes

## Summary

In this study, the researchers delve into the dynamics of transformer weight magnitudes and their relationship to training-induced movement. They discover that a two-parameter Weibull distribution effectively summarizes the weight magnitudes, with a stable shape parameter $k \approx 1.2$ across layers and models. The scale parameter $\lambda$ plays a crucial role in capturing training-induced changes in weight magnitudes. By examining the bigram conditional entropy $D = H(\text{next} \mid \text{prev})$ before training, the researchers uncover a learning-rate-conditioned law that governs how the scale parameter $\lambda$ grows. This law is expressed as $\lambda^2 - \lambda_0^2 = C_0(\eta) + C_1(\eta)(H_r - D)^{0.59}$, where $H_r$ represents a matched-budget shuffle baseline. The convex exponent of 0.59 is inherited from an independently measured data-side saturation relation, providing insight into the growth curve without direct fitting. Further analysis reveals that after removing per-learning-rate coefficients, multiple runs with varying learning rates collapse onto $(H_r - D)^{0.59}$ with high accuracy ($R^2 = 0.941$). This forward predictive model successfully recovers held-out within-family weight growth with only a 5.7% relative error, showcasing its effectiveness as a predictor. The study also explores classical metrics and estimator saturation, highlighting the importance of understanding how these metrics interact with training dynamics. Replication and causal scope are discussed to emphasize that correlations along the corruption axis do not imply causal mediation but rather indicate predictive relations between pre-training data statistics and post-training weight-scale outcomes. In conclusion, this research establishes a learning-rate-conditioned law linking pre-training data predictability to Weibull weight-scale parameter growth. By bridging measurable corpus statistics with weight-scale dynamics, this study sets the foundation for a broader framework (Φ(D, R, A, H)) where data structure, redundancy, architecture, and hyperparameters can be studied systematically one dimension at a time. The findings presented in this study provide valuable insights into understanding how pre-training data properties influence weight dynamics in transformers and offer a structured approach for analyzing these complex interactions in neural network models.