# CQM: Curriculum Reinforcement Learning with a Quantized World Model

**Authors:** Seungjae Lee, Daesol Cho, Jonghae Park, H. Jin Kim  
**arXiv:** [2310.17330](https://arxiv.org/abs/2310.17330)  
**Date:** 2023-10-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.17330v1/

## Key points

- Significant advancements in Curriculum Reinforcement Learning (RL) for complex tasks
- Introduction of a novel method to automatically define a semantic goal space and suggest curriculum goals within it
- Utilization of vector quantized-variational autoencoders (VQ-VAE) for discretizing continuous observations and establishing temporal relations through a graph
- Recommendation of uncertainty and temporal distance-aware curriculum goals that gradually converge towards final objectives
- Outperformance of existing state-of-the-art curriculum RL techniques in terms of data efficiency and performance, especially with ego-centric visual inputs
- Proposal of an approach leveraging a quantized world model to provide calibrated curriculum goals to agents, leading to significant performance improvements across various tasks

## Summary

, , , , 

In recent years, significant advancements have been made in the field of Curriculum Reinforcement Learning (RL) for tackling complex tasks. This has been achieved by proposing sequences of surrogate tasks. However, traditional approaches often struggle when generating curriculum goals in high-dimensional spaces, leading to a reliance on manually specified goal spaces. To address this limitation and enhance the scalability of curricula, a novel method has been introduced that automatically defines a semantic goal space containing crucial information for the curriculum process and suggests curriculum goals within it. This innovative approach involves discretizing continuous observations through vector quantized-variational autoencoders (VQ-VAE) and establishing temporal relations between these discretized observations using a graph. Additionally, the method recommends uncertainty and temporal distance-aware curriculum goals that gradually converge towards final objectives across the automatically composed goal space. By demonstrating its efficacy in enabling efficient explorations within uninformed environments solely based on raw goal examples, this method outperforms existing state-of-the-art curriculum RL techniques in terms of data efficiency and performance across various goal-reaching tasks, including those involving ego-centric visual inputs. Furthermore, to overcome the challenges associated with manually designing semantic goal spaces and suggesting appropriate curriculum goals to agents for complex control tasks, an approach has been proposed that leverages a quantized world model constructed from decoded embedding vectors of the discretization bottleneck. By restoring relationships between these vectors while considering uncertainty and temporal distance factors, CQM is capable of providing calibrated curriculum goals to agents. Experimental results showcase significant performance improvements across vision-based goal-reaching tasks and state-based tasks compared to traditional methods, thereby mitigating performance degradation in scenarios where manually specified goal spaces are absent.