# Hierarchical Policy for Non-prehensile Multi-object Rearrangement with Deep Reinforcement Learning and Monte Carlo Tree Search

**Authors:** Fan Bai, Fei Meng, Jianbang Liu, Jiankun Wang, Max Q. -H. Meng  
**arXiv:** [2109.08973](https://arxiv.org/abs/2109.08973)  
**Date:** 2021-09-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2109.08973v1/

## Key points

- Hierarchical policy for non-prehensile multi-object rearrangement (NPMO)
- Complexity of NPMO task due to considering object reach and order of movement
- High-level policy: Monte Carlo Tree Search (MCTS) algorithm with a designed policy network
- Low-level policy: Robot plans paths using path primitives instead of single-step discrete actions
- Experimental results show higher success rates, fewer steps, and shorter path lengths compared to state-of-the-art approaches
- Contributions: Modeling and solving NPMO with hierarchical policy, high-level MCTS policy accelerated by a trained policy network, low-level policy using path primitives
- Effective solution combining deep reinforcement learning techniques with Monte Carlo Tree Search

## Summary

This work presents a hierarchical policy for addressing the task of non-prehensile multi-object rearrangement (NPMO). NPMO involves planning feasible paths and transferring multiple objects to their predefined target poses without grasping. The complexity of this task is deepened by the need to consider how each object reaches its target and the order in which objects are moved. To tackle these challenges, the authors propose a hierarchical policy that divides and conquers the problem. In the high-level policy, a Monte Carlo Tree Search (MCTS) algorithm efficiently searches for the optimal rearrangement sequence among multiple objects. This search is guided by a designed policy network, which benefits from both imitation learning and reinforcement learning. The MCTS-based approach allows for strong long-term decision-making capabilities. In the low-level policy, the robot plans paths using path primitives, which are basic motion sequences used to push objects towards their goal poses. Unlike previous approaches that use single-step discrete actions, the proposed method reduces the depth and width of the search tree by utilizing these path primitives. Experimental results demonstrate that the proposed method achieves higher success rates, requires fewer steps, and has shorter path lengths compared to state-of-the-art approaches. The contributions of this work include modeling and solving NPMO with a hierarchical policy; proposing a high-level MCTS policy accelerated by a policy network trained with imitation and reinforcement learning; designing a low-level policy that plans paths using path primitives; achieving higher success rates; requiring fewer steps; and having shorter path lengths compared to state-of-the art approaches. Overall, this work provides an effective solution for non prehensile multi object rearrangement tasks by combining deep reinforcement learning techniques with Monte Carlo Tree Search.