# Data-Efficient Hierarchical Reinforcement Learning

**Authors:** Ofir Nachum, Shane Gu, Honglak Lee, Sergey Levine  
**arXiv:** [1805.08296](https://arxiv.org/abs/1805.08296)  
**Date:** 2018-05-21  
**Summary page:** https://www.summarizepaper.com/en/paper/1805.08296v1/

## Key points

- Hierarchical reinforcement learning (HRL) is a promising approach for complex tasks
- Existing HRL methods have limitations such as task-specific design and on-policy training
- This paper presents a study on developing general and efficient HRL algorithms
- The goal is to create algorithms that don't rely on additional assumptions and can be used with modest amounts of interaction samples
- Lower-level controllers are supervised with goals learned automatically by higher-level controllers, allowing for more flexibility in solving different tasks without manual design
- Off-policy experience is suggested for both higher and lower-level training, but changes in lower-level behaviors affect the action space for the higher-level policy
- An off-policy correction method is introduced to overcome this challenge
- The resulting HRL agent called HIRO learns both higher and lower-level policies using significantly fewer environment interactions compared to on-policy algorithms
- Experimental results show that HIRO can learn complex behaviors with only a few million samples equivalent to a few days of real-time interaction
- HIRO outperforms state-of-the-art techniques in comparisons with previous HRL methods
- This research contributes to advancing HRL by providing a general and efficient approach applicable to real-world problems like robotic control.

## Summary

In the field of reinforcement learning (RL), hierarchical reinforcement learning (HRL) has emerged as a promising approach to tackle complex tasks. However, most existing HRL methods have limitations such as requiring task-specific design and on-policy training, making them challenging to apply in real-world scenarios. To address these limitations, this paper presents a study on developing general and efficient HRL algorithms. The goal is to create algorithms that do not rely on additional assumptions beyond standard RL algorithms and can be used with modest amounts of interaction samples, making them suitable for real-world problems like robotic control. To achieve generality, the authors propose a scheme where lower-level controllers are supervised with goals that are learned and proposed automatically by the higher-level controllers. This allows for more flexibility in solving different tasks without the need for manual design. Additionally, to improve efficiency, the authors suggest using off-policy experience for both higher and lower-level training. However, this poses a challenge because changes in lower-level behaviors affect the action space for the higher-level policy. To overcome this challenge, an off-policy correction method is introduced. By leveraging recent advances in off-policy model-free RL, the resulting HRL agent called HIRO is able to learn both higher- and lower-level policies using significantly fewer environment interactions compared to on-policy algorithms. Experimental results demonstrate that HIRO can learn highly complex behaviors for simulated robots with only a few million samples equivalent to a few days of real-time interaction. Comparisons with previous HRL methods show that HIRO outperforms state-of-the-art techniques. Overall, this research contributes to advancing HRL by providing a general and efficient approach that can be applied to real-world problems like robotic control.