# DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation

**Authors:** Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, Yizhe Zhang  
**arXiv:** [2506.20639](https://arxiv.org/abs/2506.20639)  
**Date:** 2025-06-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2506.20639v1/

## Key points

- Diffusion large language models (dLLMs) are explored as alternatives to autoregressive models for code generation
- dLLMs offer global planning and iterative refinement features that are beneficial in coding tasks
- Authors conduct a systematic investigation into denoising processes and reinforcement learning methods of dLLMs in coding
- A 7B dLLM named "DiffuCoder" is trained on a massive dataset of 130B tokens of code for analysis
- Key differences between dLLMs and autoregressive models are uncovered, including the ability to determine causality without semi-autoregressive decoding
- Increasing sampling temperature diversifies token choices and alters generation order, creating a rich search space for reinforcement learning rollouts
- A novel sampling scheme called "coupled-GRPO" is proposed for reinforcement learning training to improve performance on code generation benchmarks
- DiffuCoder's performance achieves a +4.4% improvement on EvalPlus while reducing reliance on autoregressive causality during decoding
- Deeper insights into dLLM generation mechanics are provided, along with an effective diffusion-native RL training framework
- Practical considerations such as faster generation speeds with diffusion models compared to autoregressive ones are discussed

## Summary

, , , , 

In their paper titled "DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation," authors Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang explore the potential of diffusion large language models (dLLMs) as alternatives to autoregressive models for code generation. The denoising capabilities of dLLMs operating over the entire sequence offer global planning and iterative refinement features that are particularly beneficial in coding tasks. Despite this promise, the training and inference mechanisms for dLLMs in coding remain under-explored. To shed light on the decoding behavior of dLLMs and maximize their effectiveness in coding tasks, the authors conduct a systematic investigation into their denoising processes and reinforcement learning methods. They train a 7B dLLM named "DiffuCoder" on a massive dataset of 130B tokens of code to serve as a testbed for their analysis. Through their study, they uncover key differences between dLLMs and autoregressive models: firstly, dLLMs have the ability to determine the level of causality in their generation process without relying on semi-autoregressive decoding; secondly, increasing the sampling temperature not only diversifies token choices but also alters their generation order, creating a rich search space for reinforcement learning rollouts. For reinforcement learning training, the authors propose a novel sampling scheme called "coupled-GRPO" to reduce variance in token log-likelihood estimates and maintain training efficiency. This approach significantly improves DiffuCoder's performance on code generation benchmarks (achieving a +4.4% improvement on EvalPlus) while reducing reliance on autoregressive causality during decoding. Overall, this work provides deeper insights into the mechanics of dLLM generation and presents an effective diffusion-native RL training framework. Additionally, related work in text diffusion models is discussed with early explorations based on continuous space evolving into discrete diffusion models. The authors also introduce practical considerations such as faster generation speeds with diffusion models compared to autoregressive ones. Furthermore, within the context provided by section 3 detailing DiffuCoder's architecture and design principles aimed at enhancing code correctness and quality are highlighted as essential components of this research endeavor.