# Energy-Based Transformers are Scalable Learners and Thinkers

**Authors:** Alexi Gladstone, Ganesh Nanduru, Md Mofijul Islam, Peixuan Han, Hyeonjeong Ha, Aman Chadha, Yilun Du, Heng Ji, Jundong Li, Tariq Iqbal  
**arXiv:** [2507.02092](https://arxiv.org/abs/2507.02092)  
**Date:** 2025-07-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2507.02092v1/

## Key points

- Inference-time computation techniques inspired by human System 2 Thinking are being used to enhance model performance in machine learning.
- Energy-Based Transformers (EBTs) are a novel class of Energy-Based Models (EBMs) designed to assign an energy value to each input-candidate prediction pair, enabling predictions via gradient descent-based energy minimization until convergence.
- EBTs exhibit accelerated scalability across both discrete (text) and continuous (visual) modalities during training, achieving scaling rates up to 35% higher concerning data volume, batch size, parameters, FLOPs, and model depth compared to other methodologies.
- During inference scenarios, EBTs show a notable 29% enhancement in System 2 Thinking performance on language-oriented tasks compared to other models and outperform Diffusion Transformers in image denoising while requiring fewer forward passes.
- Despite comparable or inferior pretraining performance when compared with existing models, EBTs demonstrate superior results across various downstream tasks, indicating a propensity for generalization surpassing conventional approaches.

## Summary

, , , , 


In the realm of machine learning, inference-time computation techniques inspired by human System 2 Thinking have gained traction for enhancing model performance. However, many existing approaches are limited in scope and tailored to specific modalities or problem domains. Some methods also require additional supervision or training beyond unsupervised pretraining. This raises the fundamental question: can we generalize System 2 Thinking methodologies to cultivate models that autonomously learn to think through unsupervised means? Remarkably, the answer is affirmative. Enter Energy-Based Transformers (EBTs), a novel class of Energy-Based Models (EBMs) designed to assign an energy value to each input-candidate prediction pair. This framework enables predictions via gradient descent-based energy minimization until convergence. Across both discrete (text) and continuous (visual) modalities, EBTs exhibit accelerated scalability compared to other methodologies during training. , , , , 

Achieving scaling rates up to 35% higher concerning data volume, batch size, parameters, FLOPs (floating-point operations per second), and model depth underscores the efficiency of EBTs in learning tasks. During inference scenarios, EBTs showcase a notable 29% enhancement in System 2 Thinking performance on language-oriented tasks compared to other models. Furthermore, EBTs outperform Diffusion Transformers in image denoising while requiring fewer forward passes. Notably, EBTs demonstrate superior results across various downstream tasks despite comparable or inferior pretraining performance when juxtaposed with existing models. This trend suggests that EBTs possess a propensity for generalization surpassing conventional approaches. In conclusion, Energy-Based Transformers represent a promising paradigm shift in augmenting both learning and thinking capabilities within machine learning models. The fusion of unsupervised learning principles with explicit verification mechanisms propels EBTs towards heightened efficiency and efficacy across diverse modalities and problem domains.