# d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning

**Authors:** Siyan Zhao, Devaansh Gupta, Qinqing Zheng, Aditya Grover  
**arXiv:** [2504.12216](https://arxiv.org/abs/2504.12216)  
**Date:** 2025-04-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2504.12216v1/

## Key points

- Recent large language models (LLMs) have strong reasoning capabilities
- Online reinforcement learning (RL) benefits LLMs, especially in left-to-right autoregressive (AR) generation
- Diffusion-based large language models (dLLMs) generate text in a coarse-to-fine manner
- dLLMs show competitive performance compared to AR models but it's unclear if they can leverage recent advances in reasoning
- The d1 framework adapts pre-trained masked dLLMs into reasoning models through supervised fine-tuning and RL
- Techniques like masked SFT and diffu-GRPO RL algorithm are used to enhance reasoning in pretrained dLLMs
- Empirical studies show that d1 framework yields the best performance and enhances state-of-the-art dLLMs
- Advancements in scaling diffusion language models include masked diffusion as a specific instance of discrete diffusion
- Efforts are being made to address challenges related to scalability and discretization for further improvement in reasoning abilities

## Summary

Recent large language models (LLMs) have demonstrated strong reasoning capabilities that benefit from online reinforcement learning (RL), particularly within the left-to-right autoregressive (AR) generation paradigm. However, non-autoregressive paradigms based on diffusion generate text in a coarse-to-fine manner, presenting an alternative approach. Recent diffusion-based large language models (dLLMs) have shown competitive language modeling performance compared to their AR counterparts but it is unclear if they can leverage recent advances in LLM reasoning. In response to this gap, the d1 framework has been proposed to adapt pre-trained masked dLLMs into reasoning models through a combination of supervised fine-tuning (SFT) and RL. This framework introduces techniques aimed at enhancing reasoning in pretrained dLLMs, including a masked SFT technique to distill knowledge and instill self-improvement behavior from existing datasets, as well as a novel critic-free policy-gradient based RL algorithm called diffu-GRPO. Empirical studies have been conducted to evaluate the performance of different post-training recipes on various mathematical and logical reasoning benchmarks. The results indicate that d1 yields the best performance and significantly enhances the performance of state-of-the-art dLLMs. Additionally, advancements in scaling diffusion language models have been explored, with masked diffusion established as a specific instance of discrete diffusion. Efforts are being made to address challenges related to scalability and discretization in order to further improve reasoning abilities in large diffusion language models through reinforcement learning strategies.