Go-Explore: a New Approach for Hard-Exploration Problems

AI-generated keywords: Reinforcement learning Exploration Atari games Go-Explore algorithm Hard-exploration problems

AI-generated Key Points

The license of the paper does not allow us to build upon its content and the key points are generated using the paper metadata rather than the full article.

  • Reinforcement learning faces obstacles in intelligent exploration, especially in scenarios with scarce or misleading rewards.
  • Atari games Montezuma's Revenge and Pitfall are benchmarks for hard-exploration domains due to their difficulty.
  • Existing reinforcement learning algorithms using intrinsic motivation have fallen short on both games.
  • Go-Explore algorithm leverages key principles like remembering visited states, prioritizing promising states, and solving simulated environments through methods like determinism and imitation learning.
  • Go-Explore has significantly improved performance on hard-exploration problems, achieving remarkable results on Montezuma's Revenge and Pitfall.
  • With human-provided domain knowledge, Go-Explore surpasses expectations by achieving "superhuman" performance on Montezuma's Revenge and exceeding expert human performance on Pitfall.
  • Go-Explore generates high-performing demonstrations automatically and cost-effectively, surpassing traditional imitation learning approaches where humans provide solution demonstrations.
  • The success of Go-Explore opens new research avenues for enhancing capabilities and integrating insights into current reinforcement learning algorithms.
Also access our AI generated: Comprehensive summary, Lay summary, Blog-like article; or ask questions about this paper to our AI assistant.

Authors: Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, Jeff Clune

37 pages, 14 figures

Abstract: A grand challenge in reinforcement learning is intelligent exploration, especially when rewards are sparse or deceptive. Two Atari games serve as benchmarks for such hard-exploration domains: Montezuma's Revenge and Pitfall. On both games, current RL algorithms perform poorly, even those with intrinsic motivation, which is the dominant method to improve performance on hard-exploration domains. To address this shortfall, we introduce a new algorithm called Go-Explore. It exploits the following principles: (1) remember previously visited states, (2) first return to a promising state (without exploration), then explore from it, and (3) solve simulated environments through any available means (including by introducing determinism), then robustify via imitation learning. The combined effect of these principles is a dramatic performance improvement on hard-exploration problems. On Montezuma's Revenge, Go-Explore scores a mean of over 43k points, almost 4 times the previous state of the art. Go-Explore can also harness human-provided domain knowledge and, when augmented with it, scores a mean of over 650k points on Montezuma's Revenge. Its max performance of nearly 18 million surpasses the human world record, meeting even the strictest definition of "superhuman" performance. On Pitfall, Go-Explore with domain knowledge is the first algorithm to score above zero. Its mean score of almost 60k points exceeds expert human performance. Because Go-Explore produces high-performing demonstrations automatically and cheaply, it also outperforms imitation learning work where humans provide solution demonstrations. Go-Explore opens up many new research directions into improving it and weaving its insights into current RL algorithms. It may also enable progress on previously unsolvable hard-exploration problems in many domains, especially those that harness a simulator during training (e.g. robotics).

Submitted to arXiv on 30 Jan. 2019

Ask questions about this paper to our AI assistant

You can also chat with multiple papers at once here.

The license of the paper does not allow us to build upon its content and the AI assistant only knows about the paper metadata rather than the full article.

AI assistant instructions?

Results of the summarizing process for the arXiv paper: 1901.10995v1

This paper's license doesn't allow us to build upon its content and the summarizing process is here made with the paper's metadata rather than the article.

Reinforcement learning is a challenging field that faces significant obstacles when it comes to intelligent exploration. This is especially true in scenarios where rewards are scarce or misleading. Two Atari games, Montezuma's Revenge and Pitfall, have been used as benchmarks for hard-exploration domains due to their difficulty. Despite the use of intrinsic motivation in current reinforcement learning algorithms to improve performance in these environments, existing approaches have fallen short on both games. To address this limitation, a novel algorithm called Go-Explore has been introduced. This innovative approach leverages several key principles to enhance exploration efficiency: (1) remembering previously visited states, (2) prioritizing the return to promising states without further exploration before initiating new exploratory actions, and (3) solving simulated environments through various methods such as introducing determinism and robustification through imitation learning. The combination of these principles has resulted in a significant improvement in performance on hard-exploration problems. Notably, Go-Explore has achieved remarkable results on Montezuma's Revenge by scoring a mean of over 43k points – nearly quadrupling the previous state-of-the-art performance. When augmented with human-provided domain knowledge, Go-Explore surpasses expectations by achieving a mean score of over 650k points on Montezuma's Revenge and even outperforms the human world record with a maximum performance close to 18 million points – meeting the criteria for "superhuman" performance. On Pitfall – a notoriously challenging game – Go-Explore with domain knowledge becomes the first algorithm to score above zero and exceeds expert human performance with an impressive mean score of almost 60k points. Furthermore, due to its ability to generate high-performing demonstrations automatically and cost-effectively, Go-Explore surpasses traditional imitation learning approaches where humans provide solution demonstrations. The success of Go-Explore not only opens up new avenues for research into enhancing its capabilities and integrating its insights into current reinforcement learning algorithms, but also holds promise for addressing previously unsolvable hard-exploration problems across various domains – particularly those that utilize simulators during training, such as robotics. In conclusion, with its groundbreaking achievements and potential impact on advancing exploration strategies in reinforcement learning, Go-Explore represents a significant breakthrough in tackling complex challenges within this field.
Created on 11 Aug. 2024

Assess the quality of the AI-generated content by voting

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.

The license of this specific paper does not allow us to build upon its content and the summarizing tools will be run using the paper metadata rather than the full article. However, it still does a good job, and you can also try our tools on papers with more open licenses.

Similar papers summarized with our AI tools

Navigate through even more similar papers through a

tree representation

Look for similar papers (in beta version)

By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.