# Deep Reinforcement Learning for Dialogue Generation

**Authors:** Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky  
**arXiv:** [1606.01541](https://arxiv.org/abs/1606.01541)  
**Date:** 2016-06-05  
**Summary page:** https://www.summarizepaper.com/en/paper/1606.01541v1/

## Key points

- Recent neural models of dialogue generation lack foresight and only predict utterances one at a time
- Traditional NLP models have incorporated reinforcement learning to address this limitation
- The authors propose a deep reinforcement learning approach to model future reward in chatbot dialogue
- The proposed RL model rewards sequences with three important conversational properties: informativity, coherence, and ease of answering
- Evaluation is done using human judgments and automatic metrics such as conversation length and diversity
- The RL model with dialogue simulation achieves the best evaluation score in terms of sustained conversations between virtual agents
- The algorithm generates more interactive responses and fosters longer conversations compared to other models
- Diversity is assessed by calculating the number of distinct unigrams and bigrams in generated responses
- This work represents an important step towards developing a neural conversational model that considers long-term success in dialogues

## Summary

Recent neural models of dialogue generation have shown promise in generating responses for conversational agents. However, these models often lack foresight and only predict utterances one at a time without considering their impact on future outcomes. To address this limitation, traditional NLP models of dialogue have incorporated reinforcement learning to model the future direction of a conversation. In this paper, the authors propose a deep reinforcement learning approach to model future reward in chatbot dialogue. They use policy gradient methods to reward sequences that exhibit three important conversational properties: informativity (non-repetitive turns), coherence, and ease of answering. The authors evaluate their model using both human judgments and automatic metrics such as conversation length and diversity. They then compare their proposed RL model with standard SEQ2SEQ models and a mutual information model. The results show that the RL model with dialogue simulation achieves the best evaluation score in terms of sustained conversations between virtual agents. Additionally, the proposed algorithm generates more interactive responses and fosters longer conversations compared to other models. The authors also assess diversity by calculating the number of distinct unigrams and bigrams in generated responses. Overall, this work represents an important step towards developing a neural conversational model that considers long-term success in dialogues. By integrating deep reinforcement learning with traditional NLP techniques, the authors demonstrate improved performance in generating coherent and engaging dialogues for chatbot systems.