# Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization

**Authors:** Jihwan Jeong, Xiaoyu Wang, Michael Gimelfarb, Hyunwoo Kim, Baher Abdulhai, Scott Sanner  
**arXiv:** [2210.03802](https://arxiv.org/abs/2210.03802)  
**Date:** 2022-10-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2210.03802v1/

## Key points

- Offline reinforcement learning (RL) involves learning an effective policy from a fixed batch of data
- Model-based approaches are advantageous in the offline setting as they can leverage a learned model of the environment
- Existing model-based methods often underperform due to compounding estimation errors in the learned model
- The authors propose a methodology called conservative Bayesian model-based value expansion for offline policy optimization (CBOP)
- CBOP balances the use of model-free and model-based estimates based on their uncertainties
- CBOP promotes conservatism by taking a lower bound on the Bayesian posterior value estimate
- CBOP significantly outperforms previous state-of-the-art model-based approaches such as MOPO, MOReL, and COMBO
- CBOP achieves state-of-the-art performance on 11 out of 18 benchmark datasets in continuous control tasks

## Summary

The paper titled "Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization" addresses the problem of offline reinforcement learning (RL), which involves learning an effective policy from a fixed batch of data collected by following a behavior policy. The authors highlight that model-based approaches are particularly advantageous in the offline setting because they can leverage a learned model of the environment to extract more learning signals from the logged dataset. However, existing model-based methods often underperform compared to model-free counterparts due to compounding estimation errors in the learned model. To address this limitation, the authors emphasize the importance of understanding when to trust the learned model and when to rely on model-free estimates, as well as how to act conservatively with respect to both. In response, they propose a novel methodology called conservative Bayesian model-based value expansion for offline policy optimization (CBOP). This approach balances the use of model-free and model-based estimates during the policy evaluation step based on their epistemic uncertainties. It also promotes conservatism by taking a lower bound on the Bayesian posterior value estimate. The authors evaluate CBOP on standard D4RL continuous control tasks and compare its performance against previous state-of-the-art model-based approaches such as MOPO, MOReL, and COMBO. The results demonstrate that CBOP significantly outperforms these methods, achieving improvements of 116.4%, 23.2%, and 23.7% respectively. Additionally, CBOP achieves state-of-the-art performance on 11 out of 18 benchmark datasets while performing comparably on the remaining datasets. Overall, this paper introduces an elegant and simple methodology called CBOP for improving offline RL through conservative Bayesian modeling techniques. The experimental results highlight its effectiveness in surpassing existing approaches and achieving superior performance across various tasks and datasets in continuous control settings.