# A Survey of Generalisation in Deep Reinforcement Learning

**Authors:** Robert Kirk, Amy Zhang, Edward Grefenstette, Tim Rocktäschel  
**arXiv:** [2111.09794](https://arxiv.org/abs/2111.09794)  
**Date:** 2021-11-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2111.09794v1/

## Key points

- Generalisation in deep Reinforcement Learning (RL) aims to produce RL algorithms that can adapt and perform well in novel, unseen situations at deployment time.
- It is crucial for deploying reinforcement learning algorithms in real-world scenarios where the environment is diverse, dynamic, and unpredictable.
- The survey titled "A Survey of Generalisation in Deep Reinforcement Learning" provides an overview of this emerging field.
- The authors introduce a unifying formalism and terminology to discuss different generalisation problems.
- Avoiding overfitting to training environments and policies that can generalize effectively are highlighted as important factors.
- Existing benchmarks for generalisation are categorized and current methods used to tackle the problem are examined.
- Various approaches and techniques employed in deep RL algorithms to enhance generalization capabilities are explored.
- The current state of the field is critically evaluated, with recommendations for future work such as fast online adaptation and addressing RL-specific challenges.
- Relying solely on procedural content generation approaches for benchmark design is argued against, proposing exploration of underexplored problem settings like offline RL generalisation and reward-function variation when constructing benchmarks.
- The survey provides a comprehensive overview of the nascent field of generalisation in deep RL, offering valuable insights into different aspects of generalisation problems while highlighting key areas for future research and development.

## Summary

The study of generalisation in deep Reinforcement Learning (RL) aims to produce RL algorithms that can adapt and perform well in novel, unseen situations at deployment time. This is crucial for deploying reinforcement learning algorithms in real-world scenarios where the environment is diverse, dynamic, and unpredictable. In their survey titled "A Survey of Generalisation in Deep Reinforcement Learning," authors Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel provide an overview of this emerging field. The authors begin by introducing a unifying formalism and terminology to discuss different generalisation problems, building upon previous works. They emphasize the importance of avoiding overfitting to training environments and highlight the need for policies that can generalize effectively. Next, the authors categorize existing benchmarks for generalisation and examine current methods used to tackle the generalisation problem. They explore various approaches and techniques employed in deep RL algorithms to enhance generalization capabilities. In addition to discussing existing research, the authors critically evaluate the current state of the field. They offer recommendations for future work such as fast online adaptation and addressing RL-specific challenges. The authors argue against relying solely on procedural content generation approaches for benchmark design as it may hinder progress in generalisation. Furthermore, they propose exploring underexplored problem settings like offline RL generalisation and reward-function variation when constructing benchmarks so researchers can gain insights into how RL algorithms perform in these less-explored scenarios. Overall, this survey provides a comprehensive overview of the nascent field of generalisation in deep RL. It offers valuable insights into different aspects of generalisation problems while highlighting key areas for future research and development that could further improve performance on real-world tasks.