# Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning

**Authors:** Fan-Ming Luo, Tian Xu, Xingchen Cao, Yang Yu  
**arXiv:** [2310.05422](https://arxiv.org/abs/2310.05422)  
**Date:** 2023-10-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.05422v1/

## Key points

- Accurate dynamics models are crucial in offline reinforcement learning
- Existing models struggle to generalize to unseen transitions
- Dynamics reward is a key factor that remains consistent across transitions and improves generalization
- Reward-consistent dynamics models maximize the dynamics reward derived from the data
- MOREC (Model-based Offline reinforcement learning with Reward Consistency) is a method that integrates into previous offline model-based reinforcement learning methods
- MOREC learns a generalizable dynamics reward function from offline data and uses it as a transition filter
- The dynamics model selects transitions with the highest dynamics reward value
- MOREC demonstrates strong generalization ability on synthetic tasks and recovers distant unseen transitions
- MOREC outperforms previous state-of-the-art methods on D4RL and NeoRL tasks, achieving improvements of 4.6% and 25.9% respectively
- MOREC achieves above 95% online RL performance in several D4RL and NeoRL tasks, highlighting its potential for real-world applications.

## Summary

In the field of offline reinforcement learning, it is crucial to have an accurate dynamics model. However, existing models that are learned by fitting historical transitions often struggle to generalize to unseen transitions. In this study, the authors identify a key factor called dynamics reward that remains consistent across transitions and can improve generalization. They propose the concept of reward-consistent dynamics models, where any trajectory generated by the dynamics model should maximize the dynamics reward derived from the data. To implement this idea, they introduce a method called MOREC (Model-based Offline reinforcement learning with Reward Consistency). MOREC can be seamlessly integrated into previous offline model-based reinforcement learning (MBRL) methods. It learns a generalizable dynamics reward function from offline data and uses it as a transition filter in any offline MBRL method. When generating transitions, the dynamics model generates a batch of transitions and selects the one with the highest dynamics reward value. The authors demonstrate the effectiveness of MOREC on both synthetic tasks and 21 offline tasks in D4RL and NeoRL benchmarks. On synthetic tasks, MOREC exhibits strong generalization ability and can recover distant unseen transitions. On D4RL and NeoRL tasks, MOREC significantly outperforms previous state-of-the-art methods, achieving improvements of 4.6% on D4RL tasks and 25.9% on NeoRL tasks. Notably, MOREC is the first method that achieves above 95% online RL performance in 6 out of 12 D4RL tasks and 3 out of 9 NeoRL tasks. This highlights its potential for practical applications in real-world scenarios.