# Unsupervised-to-Online Reinforcement Learning

**Authors:** Junsu Kim, Seohong Park, Sergey Levine  
**arXiv:** [2408.14785](https://arxiv.org/abs/2408.14785)  
**Date:** 2024-08-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2408.14785v1/

## Key points

- Offline-to-online RL is a method to train policies using offline RL and then fine-tune them with online RL
- Limitations of traditional offline-to-online RL include the need for domain-specific offline RL pre-training and fragility in practical applications
- Unsupervised-to-online RL (U2O RL) is proposed as an improved alternative, replacing supervised offline RL with unsupervised offline RL
- U2O RL allows reuse of a single pre-trained model across multiple tasks and leads to enhanced representations, superior performance, and stability
- A comprehensive recipe bridges unsupervised offline skill-based policy pre-training with supervised online fine-tuning to operationalize U2O RL
- Empirical validation across nine diverse environments shows U2O RL achieves robust performance matching or surpassing existing approaches
- Leveraging a single pre-trained model for various tasks highlights the versatility and efficiency of U2O RL

## Summary

In the realm of reinforcement learning (RL), the concept of offline-to-online RL has gained traction as a method to train policies using offline RL and then fine-tune them with online RL. This offers a promising avenue for data-driven decision-making. However, this approach has limitations such as the need for domain-specific offline RL pre-training for each task and its tendency to be fragile in practical applications. To address these challenges, a new framework called unsupervised-to-online RL (U2O RL) is proposed as an improved alternative to traditional offline-to-online RL. U2O RL distinguishes itself by replacing domain-specific supervised offline RL with unsupervised offline RL. This not only allows for the reuse of a single pre-trained model across multiple downstream tasks but also facilitates the learning of enhanced representations, often leading to superior performance and stability compared to supervised offline-to-online RL methods. To operationalize U2O RL, a comprehensive recipe is presented that bridges task-agnostic unsupervised offline skill-based policy pre-training with supervised online fine-tuning. Empirical validation through experiments conducted across nine diverse environments—both state-based and pixel-based—demonstrates the efficacy of U2O RL in achieving robust performance that either matches or surpasses existing offline-to-online RL approaches. Furthermore, the ability to leverage a single pre-trained model for various downstream tasks underscores the versatility and efficiency of U2O RL in real-world applications. The work is authored by Junsu Kim, Seohong Park, and Sergey Levine, highlighting their contributions to advancing the field of reinforcement learning through innovative methodologies like U2O RL.