# Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models

**Authors:** Mohammad Emtiyaz Khan, Wu Lin  
**arXiv:** [1703.04265](https://arxiv.org/abs/1703.04265)  
**Date:** 2017-03-13  
**Summary page:** https://www.summarizepaper.com/en/paper/1703.04265v1/

## Key points

- Variational inference is a popular technique for approximating complex probability distributions in statistical modeling.
- Dealing with models that contain both conjugate and non-conjugate terms becomes computationally challenging.
- Conjugate models have easier computations as the prior and posterior distributions belong to the same family.
- Non-conjugate models require more sophisticated methods as they involve priors and posteriors from different families.
- Existing methods designed for conjugate models are efficient but struggle with non-conjugate terms.
- Stochastic-gradient methods can handle non-conjugate terms but often ignore the conjugate structure, resulting in slow convergence.
- The paper proposes a new algorithm called Conjugate-computation Variational Inference (CVI) that combines conjugate computations and stochastic gradients.
- CVI uses conjugate computations for the conjugate terms and employs stochastic gradients for the rest of the model.
- The authors derive CVI using a stochastic mirror-descent method in the mean-parameter space and express each gradient step as a variational inference in a conjugate model.
- CVI is applicable to various models and has established convergence properties.
- Experimental results show that CVI converges faster than methods ignoring the conjugate structure of the model.
- The proposed CVI algorithm offers improved efficiency and convergence rates compared to existing methods.

## Summary

In the field of statistical modeling, variational inference is a popular technique for approximating complex probability distributions. However, it becomes computationally challenging when dealing with models that contain both conjugate and non-conjugate terms. Conjugate models are those in which the prior and posterior distributions belong to the same family, making computations easier. On the other hand, non-conjugate models involve priors and posteriors from different families, requiring more sophisticated methods. Existing methods designed specifically for conjugate models are computationally efficient but struggle to handle non-conjugate terms effectively. Conversely, stochastic-gradient methods can handle non-conjugate terms but often ignore the conjugate structure of the model, leading to slow convergence. To address this issue, this paper proposes a new algorithm called Conjugate-computation Variational Inference (CVI). The CVI algorithm combines the strengths of both approaches by using conjugate computations for the conjugate terms and employing stochastic gradients for the rest. The authors derive this algorithm by utilizing a stochastic mirror-descent method in the mean-parameter space and expressing each gradient step as a variational inference in a conjugate model. The authors demonstrate its applicability to a wide range of models and establish its convergence properties. Experimental results show that CVI converges much faster than methods that ignore the conjugate structure of the model. Overall, this paper presents an innovative solution to overcome computational challenges in variational inference by integrating conjugate computations and stochastic gradients. The proposed CVI algorithm offers improved efficiency and convergence rates compared to existing methods, making it a valuable tool for analyzing complex probabilistic models.