# The Free Transformer

**Authors:** François Fleuret  
**arXiv:** [2510.17558](https://arxiv.org/abs/2510.17558)  
**Date:** 2025-10-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2510.17558v1/

## Key points

- The Free Transformer is an extension of the decoder Transformer that incorporates random latent variables learned through a variational procedure to condition its generative process.
- Experimental evaluations show significant improvements on downstream tasks, especially on benchmarks requiring reasoning skills like HumanEval+, MBPP, and GSM8K.
- Performance of 1.5B and 8B models with varying levels of information per token is compared in tables, demonstrating enhancements in tasks like MMLU and CSQA for the 8B model with lower KL divergence.
- Graphs in the appendix illustrate performance trends during training.
- Analysis shows how the model's behavior changes at different levels of KL divergence, transitioning from behaving like a vanilla model to encoding target positions and noise before generating incorrect sequences as divergence value increases.

## Summary

The Free Transformer is an extension of the decoder Transformer that incorporates random latent variables learned through a variational procedure to condition its generative process. Experimental evaluations demonstrate significant improvements on downstream tasks, particularly on benchmarks requiring reasoning skills such as HumanEval+, MBPP, and GSM8K. The performance of 1.5B and 8B models with varying levels of information per token is compared in tables, showing enhancements in tasks like MMLU and CSQA for the 8B model with lower KL divergence. Graphs in the appendix illustrate performance trends during training. Additionally, the model's behavior at different levels of KL divergence is analyzed, showing how it transitions from behaving like a vanilla model to encoding target positions and noise before generating incorrect sequences as the divergence value increases. This detailed exploration provides insights into how the Free Transformer adapts its generative process based on the learned latent variables.