Running summarizing tools on a new article

Check the paper below, then start its synthesis.


The Mechanics of Explainability: How Exponential Attention Enables Sparse and Interpretable in-Context Learning

Yeqi Gao (Tsinghua Shenzhen International Graduate School, Tsinghua University), Erçan E. Kuruoğlu (Tsinghua Shenzhen International Graduate School, Tsinghua University)

Multidimensional Signal Processing Workshop
License: CC BY

Abstract: Post-hoc explainability methods such as LIME and SHAP approximate model behavior externally but cannot reveal the intrinsic decision-making logic of Transformers during incontext learning (ICL). We argue that Softmax attention is not merely a computational component but an inherently interpretable mechanism whose dynamics can be directly read as an explanation. Our key observation is that the exponential nonlinearity in Softmax creates a self-reinforcing feedback loop during optimization: tokens slightly more aligned with the task receive multiplicatively larger gradient updates, causing attention entropy to collapse and producing sparse, human-readable attention maps that transparently indicate which context tokens drive the prediction. We formalize this as an attention interpretability principle: under exponential dynamics with sufficient model capacity, the trained attention distribution converges to a sparse indicator over task-relevant tokens, functioning as a built-in explanation without any auxiliary interpretability module. Through controlled experiments we demonstrate that this inherent interpretability (i) emerges as a sharp phase transition during training and (ii) requires sufficient width to activate. We further argue on structural grounds that the resulting intrinsic explanations are more faithful and stable than post-hoc attribution methods. Our findings suggest that properly configured Softmax attention offers a principled alternative to external explainability tools for understanding ICL decisions.

Published on 01 Jun. 2026