# Offline Reinforcement Learning with Pseudometric Learning

**Authors:** Robert Dadashi, Shideh Rezaeifar, Nino Vieillard, Léonard Hussenot, Olivier Pietquin, Matthieu Geist  
**arXiv:** [2103.01948](https://arxiv.org/abs/2103.01948)  
**Date:** 2021-03-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2103.01948v1/

## Key points

- Offline Reinforcement Learning (RL) methods aim to learn a policy from logged transitions without interaction
- Importance of ensuring learned policy visits state-action pairs close to support of logged transitions
- Authors propose iterative procedure to learn pseudometric from logged transitions
- Pseudometric is related to bisimulation metrics and defines closeness between state-action pairs
- Convergence of iterative procedure demonstrated, extended to function approximation setting
- Pseudometric used in actor-critic algorithm called PLOff with lookup-based bonus
- Bonus encourages actor to remain close to support of logged transitions, enhancing exploration and learning effective policy
- Experiments conducted on hand manipulation and locomotion tasks, demonstrating method's performance
- Approach addresses challenges of offline RL with limited coverage of state-action space

## Summary

, , , , 

Offline Reinforcement Learning (RL) methods aim to learn a policy from logged transitions of an environment, without any interaction. However, when function approximation is involved and the state-action space of the environment has limited coverage, it becomes crucial to ensure that the learned policy visits state-action pairs that are close to the support of logged transitions. In this study, the authors propose an iterative procedure to learn a pseudometric from logged transitions. This pseudometric is closely related to bisimulation metrics and is used to define a notion of closeness between state-action pairs. The authors demonstrate the convergence of this iterative procedure and extend it to the function approximation setting. Furthermore, the authors utilize this pseudometric to introduce a new lookup-based bonus in an actor-critic algorithm called PLOff. This bonus encourages the actor to remain close, according to the defined pseudometric, to the support of logged transitions. By doing so, PLOff enhances exploration and helps in learning a more effective policy. To evaluate their method's performance, the authors conduct experiments on hand manipulation and locomotion tasks. The results demonstrate that their approach effectively addresses the challenges posed by offline RL with limited coverage of state-action space. Overall, this research contributes by proposing a novel approach for offline RL using pseudometric learning. The introduced method not only ensures exploration near logged transitions but also achieves promising results in various complex tasks such as hand manipulation and locomotion.