# Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation

**Authors:** Vincent Mai, Kaustubh Mani, Liam Paull  
**arXiv:** [2201.01666](https://arxiv.org/abs/2201.01666)  
**Date:** 2022-01-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2201.01666v1/

## Key points

- Inverse Variance Reinforcement Learning (IV-RL) is a Bayesian framework for improving sample efficiency and performance of model-free deep reinforcement learning algorithms
- IV-RL combines probabilistic ensembles and Batch Inverse Variance weighting to estimate the variance of the target and down weight uncertain samples in two complementary ways
- The authors provide a systematic analysis of sources of uncertainty in noisy supervision that occurs in RL and propose a method whereby two complementary uncertainty estimation methods account for both the Q-value and environment stochasticity to better mitigate the negative impacts of noisy supervision
- Results show significant improvement in terms of sample efficiency on discrete and continuous control tasks compared to state-of-the-art methods
- The IV-RL framework is adaptable to any model-free algorithm, including DQN and SAC, as well as model-based, active or continuous learning
- The authors encourage users to consider ethical issues related to their field of application when using their algorithm
- They also publish their carbon footprint from experiments conducted on private infrastructure with a carbon efficiency of 0.028 kgCO2eq/kWh
- To ensure reproducibility, the code used in this study along with hyperparameters are available at https://github.com/montrealrobotics/iv_rl
- The environments used (OpenAI Gym, BSuite, MBBL) are publicly accessible but may require a Mujoco license for some tasks
- Overall, IV-RL represents a significant step towards enabling DRL applications in real-world problems such as robotics where sample efficiency is crucial

## Summary

This paper presents Inverse Variance Reinforcement Learning (IV-RL), a Bayesian framework for improving the sample efficiency and performance of model-free deep reinforcement learning (RL) algorithms by leveraging uncertainty estimation. IV-RL combines probabilistic ensembles and Batch Inverse Variance weighting to estimate the variance of the target and down weight uncertain samples in two complementary ways. The authors provide a systematic analysis of sources of uncertainty in noisy supervision that occurs in RL and propose a method whereby two complementary uncertainty estimation methods account for both the Q-value and environment stochasticity to better mitigate the negative impacts of noisy supervision. The results show significant improvement in terms of sample efficiency on discrete and continuous control tasks compared to state-of-the-art methods. The IV-RL framework is adaptable to any model-free algorithm, including DQN and SAC, as well as model-based, active or continuous learning. The authors encourage users to consider ethical issues related to their field of application when using their algorithm. They also publish their carbon footprint from experiments conducted on private infrastructure with a carbon efficiency of 0.028 kgCO2eq/kWh. To ensure reproducibility, the code used in this study along with hyperparameters are available at https://github.com/montrealrobotics/iv_rl. The environments used (OpenAI Gym, BSuite, MBBL) are publicly accessible but may require a Mujoco license for some tasks. Overall, IV-RL represents a significant step towards enabling DRL applications in real-world problems such as robotics where sample efficiency is crucial.