# Dangers of Bayesian Model Averaging under Covariate Shift

**Authors:** Pavel Izmailov, Patrick Nicholson, Sanae Lotfi, Andrew Gordon Wilson  
**arXiv:** [2106.11905](https://arxiv.org/abs/2106.11905)  
**Date:** 2021-06-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2106.11905v1/

## Key points

- The paper discusses the limitations of Bayesian neural networks (BNNs) with high-fidelity approximate inference via full-batch Hamiltonian Monte Carlo under covariate shift.
- BNNs are generally robust and perform well on out-of-distribution data, but this specific inference method fails to achieve good generalization under covariate shift.
- Bayesian model averaging can be problematic in the presence of covariate shift, particularly when there are linear dependencies in input features leading to a lack of posterior contraction.
- This issue does not affect many other approximate inference procedures or classical maximum a-posteriori (MAP) training methods.
- The authors propose novel priors to enhance the robustness of BNNs against various sources of covariate shift.
- These new priors aim to improve modeling approach and account for challenges posed by linear dependencies in input features.
- The study sheds light on an important drawback of Bayesian model averaging in BNNs under covariate shift and provides insights into more resilient inference procedures.
- The proposed novel priors offer a potential solution to improve performance and robustness of BNNs in scenarios involving covariate shift.

## Summary

The paper titled "Dangers of Bayesian Model Averaging under Covariate Shift" discusses the limitations of Bayesian neural networks (BNNs) with high-fidelity approximate inference via full-batch Hamiltonian Monte Carlo when it comes to generalization under covariate shift. BNNs are generally considered a robust alternative to standard training and perform well on out-of-distribution data. However, this specific type of inference method fails to achieve good generalization under covariate shift and even underperforms classical estimation techniques. The authors delve into this surprising result and explain how a Bayesian model average can be problematic in the presence of covariate shift, particularly when there are linear dependencies in the input features that lead to a lack of posterior contraction. The study highlights that this issue does not affect many other approximate inference procedures or classical maximum a-posteriori (MAP) training methods. To address these limitations, the authors propose novel priors that enhance the robustness of BNNs against various sources of covariate shift. These new priors aim to overcome the poor generalization observed in BNNs with high-fidelity approximate inference by improving the modeling approach and accounting for the challenges posed by linear dependencies in input features. Overall, this study sheds light on an important drawback of Bayesian model averaging in BNNs under covariate shift and provides insights into why certain inference procedures are more resilient to this issue. The proposed novel priors offer a potential solution to improve the performance and robustness of BNNs in scenarios involving covariate shift.