# AlphaStar Unplugged: Large-Scale Offline Reinforcement Learning

**Authors:** Michaël Mathieu, Sherjil Ozair, Srivatsan Srinivasan, Caglar Gulcehre, Shangtong Zhang, Ray Jiang, Tom Le Paine, Richard Powell, Konrad Żołna, Julian Schrittwieser, David Choi, Petko Georgiev, Daniel Toyama, Aja Huang, Roman Ring, Igor Babuschkin, Timo Ewalds, Mahyar Bordbar, Sarah Henderson, Sergio Gómez Colmenarejo  
**arXiv:** [2308.03526](https://arxiv.org/abs/2308.03526)  
**Date:** 2023-08-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.03526v1/

## Key points

- StarCraft II is a highly challenging simulated reinforcement learning environment
- It presents unique difficulties for AI agents
- It is partially observable, stochastic, and multi-agent game with strategic planning over long time horizons and real-time low-level execution
- StarCraft II has an active professional competitive scene, making it ideal for advancing offline RL algorithms
- Blizzard has released a massive dataset of millions of games played by human players to facilitate research in this area
- The authors introduce AlphaStar Unplugged as a benchmark for offline reinforcement learning in StarCraft II
- The benchmark includes a subset of Blizzard's dataset and tools to standardize an API for machine learning methods and an evaluation protocol
- Baseline agents are presented, including behavior cloning and offline variants of actor-critic and MuZero algorithms trained using only offline data from the dataset provided by Blizzard
- The authors achieve significant improvements over previously published AlphaStar behavior cloning agent with a 90% win rate
- This research contributes to advancing the state-of-the-art in AI agents for complex real-time strategy games like StarCraft II.

## Summary

StarCraft II is a highly challenging simulated reinforcement learning environment that presents unique difficulties for AI agents. It is a partially observable, stochastic, and multi-agent game with strategic planning over long time horizons and real-time low-level execution. Additionally, StarCraft II has an active professional competitive scene, making it an ideal platform for advancing offline RL algorithms. To facilitate research in this area, Blizzard has released a massive dataset of millions of games played by human players. Leveraging this dataset, the authors of the paper introduce AlphaStar Unplugged, a benchmark that sets unprecedented challenges for offline reinforcement learning. The benchmark includes a subset of Blizzard's dataset as well as tools to standardize an API for machine learning methods and an evaluation protocol. The paper also presents baseline agents for comparison including behavior cloning and offline variants of actor-critic and MuZero algorithms which are trained using only offline data from the dataset provided by Blizzard. Notably, the authors achieve significant improvements over previously published AlphaStar behavior cloning agent with a 90% win rate. In summary, this paper explores the potential of offline reinforcement learning in the context of StarCraft II by leveraging a massive dataset provided by Blizzard. The authors establish AlphaStar Unplugged as a benchmark for evaluating RL algorithms in this challenging environment and present baseline agents that demonstrate improved performance using only offline data. This research contributes to advancing the state-of-the-art in AI agents for complex real-time strategy games like StarCraft II.