# Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion

**Authors:** Boyuan Chen, Diego Marti Monso, Yilun Du, Max Simchowitz, Russ Tedrake, Vincent Sitzmann  
**arXiv:** [2407.01392](https://arxiv.org/abs/2407.01392)  
**Date:** 2024-07-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.01392v1/

## Key points

- Novel training paradigm combining next-token prediction models and full-sequence diffusion models
- Allows for variable-length generation and guides sampling towards desirable trajectories
- Offers capabilities such as rolling out sequences beyond training horizon, new sampling schemes, and improved decision-making performance
- Demonstrates flexibility in controllable sequential compositional generation
- Enables long-horizon imitation learning and robust visuomotor control in robotics applications
- Shows robustness to noisy or missing observations through Bayes filtering principles
- Competes favorably with prior approaches in time series forecasting
- Applicable in language modeling, planning, video generation, robotic manipulation, and time series forecasting

## Summary

is a novel training paradigm that combines the strengths of next-token prediction models and full-sequence diffusion models in sequence generative modeling. This approach allows for variable-length generation while guiding sampling towards desirable trajectories. It also offers additional capabilities such as rolling out sequences of continuous tokens beyond the training horizon, introducing new sampling and guiding schemes, and leveraging its variable-horizon and causal architecture for improved performance in decision-making and planning tasks. The study demonstrates the flexibility of  in controllable sequential compositional generation by modifying the sampling scheme to compose sub-sequences of observed sequences. In robotics applications,  enables long-horizon imitation learning and robust visuomotor control by incorporating memory into the latent state. This results in higher success rates compared to diffusion policy algorithms without memory. Additionally, this method shows robustness to noisy or missing observations through principles from Bayes filtering. Furthermore,  proves to be a good general-purpose sequence model in time series forecasting, competing favorably with prior diffusion and transformer-based approaches. The paper showcases the potential of  in various applications such as language modeling, planning, video generation, robotic manipulation, and time series forecasting. Overall,presents a promising approach that bridges the gap between next-token prediction models and full-sequence diffusion models for enhanced sequence generative modeling across diverse domains.