# Human-Timescale Adaptation in an Open-Ended Task Space

**Authors:**  Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Loks-Thompson, Hannah Openshaw, Jack Parker-Holder, Shreya Pathak, Nicolas Perez-Nieves  
**arXiv:** [2301.07608](https://arxiv.org/abs/2301.07608)  
**Date:** 2023-01-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2301.07608v1/

## Key points

- Training reinforcement learning (RL) agents at scale to develop a general in-context learning algorithm
- Adaptive agent (AdA) can quickly adapt to novel embodied 3D problems as efficiently as humans
- Three key components for AdA's adaptation: meta-reinforcement learning, attention-based memory architecture, and effective automated curriculum
- Insights into scaling laws associated with network size, memory length, and richness of training task distribution
- Consideration of computational costs associated with different model sizes
- Increasing memory length improves performance, particularly on the tails of the distribution
- Positive scaling of adaptation with task pool size
- Foundation for developing increasingly general and adaptive RL agents in open-ended domains
- Valuable insights into optimizing model size, memory length, and task distribution richness for efficient RL training

## Summary

In this work, the authors demonstrate the potential of training reinforcement learning (RL) agents at scale to develop a general in-context learning algorithm. They show that their adaptive agent (AdA) can quickly adapt to novel embodied 3D problems as efficiently as humans. The adaptation of AdA is achieved through three key components: meta-reinforcement learning across a diverse task distribution, a policy parameterized as a large-scale attention-based memory architecture, and an effective automated curriculum that prioritizes tasks at the frontier of the agent's capabilities. The authors provide insights into the scaling laws associated with network size, memory length and richness of the training task distribution. They also discuss the computational costs associated with different model sizes and highlight that larger models may not always be the best choice when considering compute cost. Furthermore, they investigate how performance scales with the length of AdA's memory and by examining different values for caching previous network activations they find that increasing memory length improves performance, particularly on the tails of the distribution. The authors also explore how adaptation scales with the size of the task pool and observe that median and 20th percentile adaptation scales positively with an increase in task pool size. Overall, this work lays the foundation for developing increasingly general and adaptive RL agents that excel in open-ended domains. The findings contribute valuable insights into optimizing model size, memory length and task distribution richness for efficient RL training.