# Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations

**Authors:** Lucy Farnik, Tim Lawson, Conor Houghton, Laurence Aitchison  
**arXiv:** [2502.18147](https://arxiv.org/abs/2502.18147)  
**Date:** 2025-02-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2502.18147v2/

## Key points

- Researchers aimed to enhance understanding of computations performed by LLMs beyond just their representations
- Introduced Jacobian Sparse Autoencoders (JSAEs) to induce sparsity in input and output activations as well as computational connections between them
- Devised an efficient method for computing Jacobians in LLMs, enabling considerable computational sparsity without compromising performance
- JSAEs identified semantically meaningful computational units within LLMs, such as phrases like "this text is in German"
- Pre-trained LLMs exhibit significantly more sparse computational connections compared to randomly initialized transformers, indicating learned property during training
- JSAEs offer potential for better understanding transformer operations and enhancing interpretability of deep learning models

## Summary

In the study "Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations," researchers aimed to enhance the understanding of computations performed by  beyond just their representations. While  have been effective in uncovering sparse and interpretable latent activations of LLMs, they do not specifically target sparsity in computations. To address this limitation, the researchers introduced , which not only induce sparsity in input and output activations but also in the computational connections () between them. One significant technical contribution of this research was devising an efficient method for computing Jacobians in LLMs, as traditional approaches would be computationally prohibitive due to their size. The results demonstrated that JSAEs can achieve a considerable degree of computational sparsity while maintaining LLM performance comparable to conventional SAEs. Moreover, the study showed that  serve as a reasonable proxy for computational sparsity, particularly evident in Multilayer Perceptrons (MLPs) when expressed in the JSAE basis. By analyzing "max-activating" examples of JSAEs, the researchers verified that these models can identify semantically meaningful computational units within LLMs. For instance, specific output SAE latents were found to correspond to phrases such as "this text is in German," computed based on input latents representing tokens common in German text or related to historical events like the Third Reich. Furthermore, comparisons with randomly initialized transformers revealed that pre-trained LLMs exhibit significantly more sparse , indicating that computational sparsity is a learned property during training. This contrasts with previous findings showing similar interpretability scores for SAEs on random and pre-trained transformers. The study also provided insights into how JSAEs extract information about complex learned computations and highlighted their potential for understanding transformer operations better than standard SAEs. In conclusion, "Jacobian Sparse Autoencoders" presents a novel approach to enhancing interpretability and understanding of computations within LLMs through sparsity-inducing techniques like JSAEs. The findings underscore the importance of considering not just representations but also the underlying computations in advancing our comprehension of deep learning models.