# Open-Ended Learning Leads to Generally Capable Agents

**Authors:** Open-Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, Nat McAleese, Nathalie Bradley-Schmieg, Nathaniel Wong, Nicolas Porcel, Roberta Raileanu, Steph Hughes-Fitt, Valentin Dalibard, Wojciech Marian Czarnecki  
**arXiv:** [2107.12808](https://arxiv.org/abs/2107.12808)  
**Date:** 2021-07-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2107.12808v1/

## Key points

- Authors present a novel approach to training agents for diverse tasks in a challenging environment
- Agents are trained in a multi-agent environment with competitive, cooperative, and independent games
- Proposed iterative improvement between generations of agents instead of maximizing a singular objective
- Open-ended learning process with dynamically changing training task distributions and objectives
- Agent shows remarkable capabilities, scoring rewards in every humanly solvable evaluation level
- Zero-shot generalization demonstrated in tasks like Hide and Seek, Capture the Flag, and Tag
- Emergent heuristic behaviors identified such as trial-and-error experimentation, simple tool use, option switching, and cooperation
- General capabilities enable larger-scale transfer of behavior through cheap fine-tuning

## Summary

In this work, the authors present a novel approach to training agents that can perform well across a wide range of tasks and exhibit generalization of behavior in a diverse and challenging environment. They define a universe of tasks within a multi-agent environment domain, which includes competitive, cooperative, and independent games situated in procedurally generated 3D worlds. The challenges posed to the agents in this environment are exceptionally diverse, making it difficult to measure their learning progress. To address this challenge, the authors propose an iterative notion of improvement between successive generations of agents instead of maximizing a singular objective. This allows them to quantify progress even when tasks have incomparable rewards. They demonstrate that by constructing an open-ended learning process that dynamically changes the training task distributions and objectives, they achieve consistent learning of new behaviors. The resulting agent shows remarkable capabilities, being able to score rewards in every humanly solvable evaluation level and generalize its behavior to many held-out points in the universe of tasks. Notably, the agent demonstrates zero-shot generalization in tasks such as Hide and Seek, Capture the Flag, and Tag. Through analysis and hand-authored probe tasks, the authors characterize the behavior of their agent and identify interesting emergent heuristic behaviors such as trial-and-error experimentation, simple tool use, option switching, and cooperation. Finally, they demonstrate that the general capabilities of their agent could enable larger-scale transfer of behavior through cheap fine-tuning. Overall, this work presents an innovative approach to training agents that can perform well across a wide range of tasks while exhibiting generalization in complex environments with remarkable capabilities such as zero-shot generalization for various game types like Hide and Seek or Capture The Flag. Furthermore it is capable of identifying emergent heuristic behaviors like trial-and-error experimentation or simple tool use which enables larger scale transfer through cheap fine tuning.