Bayesian Learning for Neural Networks: an algorithmic survey
Authors: Martin Magris, Alexandros Iosifidis
Abstract
The last decade witnessed a growing interest in Bayesian learning. Yet, the technicality of the topic and the multitude of ingredients involved therein, besides the complexity of turning theory into practical implementations, limit the use of the Bayesian learning paradigm, preventing its widespread adoption across different fields and applications. This self-contained survey engages and introduces readers to the principles and algorithms of Bayesian Learning for Neural Networks. It provides an introduction to the topic from an accessible, practical-algorithmic perspective. Upon providing a general introduction to Bayesian Neural Networks, we discuss and present both standard and recent approaches for Bayesian inference, with an emphasis on solutions relying on Variational Inference and the use of Natural gradients. We also discuss the use of manifold optimization as a state-of-the-art approach to Bayesian learning. We examine the characteristic properties of all the discussed methods, and provide pseudo-codes for their implementation, paying attention to practical aspects, such as the computation of the gradients
The papers around it, by date and closeness
Higher is closer to this paper; left is older. Lines join papers close to each other: groups of lines are sub-topics. Select a point to see why it is there. Back to the summary of this paper
Papers around this one
Ranked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
A Primer on Bayesian Neural Networks: Review and DebatesJulyan Arbel, Konstantinos Pitas, Mariia Vladimirova et al. · 2023 · Machine Learning · preprint, not peer reviewed · cited 60 times · AI summary available
Abstract
Neural networks have achieved remarkable performance across various problem domains, but their widespread applicability is hindered by inherent limitations such as overconfidence in predictions, lack of interpretability, and vulnerability to adversarial attacks. To address these challenges, Bayesian neural networks (BNNs) have emerged as a compelling extension of conventional neural networks, integrating uncertainty estimation into their predictive capabilities. This comprehensive primer presents a systematic introduction to the fundamental concepts of neural networks and Bayesian inference, elucidating their synergistic integration for the development of BNNs. The target audience comprises …
-
Hands-On Bayesian Neural Networks—A Tutorial for Deep Learning UsersLaurent Valentin Jospin, Hamid Laga, F. Boussaid et al. · 2020 · IEEE Computational Intelligence Magazine · cited 977 times · Not summarized yet
Abstract
Modern deep learning methods constitute incredibly powerful tools to tackle a myriad of challenging problems. However, since deep learning methods operate as black boxes, the uncertainty associated with their predictions is often challenging to quantify. Bayesian statistics offer a formalism to understand and quantify the uncertainty associated with deep neural network predictions. This tutorial provides deep learning practitioners with an overview of the relevant literature and a complete toolset to design, implement, train, use and evaluate Bayesian neural networks, i.e., stochastic artificial neural networks trained using Bayesian methods.
-
Practical Deep Learning with Bayesian PrinciplesKazuki Osawa, S. Swaroop, Mohammad Emtiyaz Khan et al. · 2019 · Neural Information Processing Systems · cited 288 times · Not summarized yet
Abstract
Bayesian methods promise to fix many shortcomings of deep learning, but they are impractical and rarely match the performance of standard methods, let alone improve them. In this paper, we demonstrate practical training of deep networks with natural-gradient variational inference. By applying techniques such as batch normalisation, data augmentation, and distributed training, we achieve similar performance in about the same number of epochs as the Adam optimiser, even on large datasets such as ImageNet. Importantly, the benefits of Bayesian principles are preserved: predictive probabilities are well-calibrated, uncertainties on out-of-distribution data are improved, and continual-learning pe…
-
DE-BNN: An evolutionary approach to Bayesian neural network posterior samplingWesley Forbes, Min Long · 2026 · Neurocomputing · Not summarized yet
-
Priors in Bayesian Deep Learning: A ReviewVincent Fortuin · 2021 · Machine Learning · cited 201 times · AI summary available
Abstract
While the choice of prior is one of the most critical parts of the Bayesian inference workflow, recent Bayesian deep learning models have often fallen back on vague priors, such as standard Gaussians. In this review, we highlight the importance of prior choices for Bayesian deep learning and present an overview of different priors that have been proposed for (deep) Gaussian processes, variational autoencoders, and Bayesian neural networks. We also outline different methods of learning priors for these models from data. We hope to motivate practitioners in Bayesian deep learning to think more carefully about the prior specification for their models and to provide them with some inspiration in…
-
Bayesian Transfer LearningPiotr M. Suder, Jason Xu, David B. Dunson · 2023 · Machine Learning · cited 17 times · AI summary available
Abstract
Transfer learning is a burgeoning concept in statistical machine learning that seeks to improve inference and/or predictive accuracy on a domain of interest by leveraging data from related domains. While the term "transfer learning" has garnered much recent interest, its foundational principles have existed for years under various guises. Prior literature reviews in computer science and electrical engineering have sought to bring these ideas into focus, primarily surveying general methodologies and works from these disciplines. This article highlights Bayesian approaches to transfer learning, which have received relatively limited attention despite their innate compatibility with the notion …
-
Variational Bayesian Learning TheoryShinichi Nakajima, Kazuho Watanabe, Masashi Sugiyama · 2019 · cited 32 times · Not summarized yet
Abstract
Variational Bayesian learning is one of the most popular methods in machine learning. Designed for researchers and graduate students in machine learning, this book summarizes recent developments in the non-asymptotic and asymptotic theory of variational Bayesian learning and suggests how this theory can be applied in practice. The authors begin by developing a basic framework with a focus on conjugacy, which enables the reader to derive tractable algorithms. Next, it summarizes non-asymptotic theory, which, although limited in application to bilinear models, precisely describes the behavior of the variational Bayesian solution and reveals its sparsity inducing mechanism. Finally, the text su…
-
A Tutorial on Learning with Bayesian NetworksD. Heckerman · 1999 · Innovations in Bayesian Networks · cited 3,655 times · Not summarized yet
Abstract
A Bayesian network is a graphical model that encodes probabilistic relationships among variables of interest. When used in conjunction with statistical techniques, the graphical model has several advantages for data analysis. One, because the model encodes dependencies among all variables, it readily handles situations where some data entries are missing. Two, a Bayesian network can be used to learn causal relationships, and hence can be used to gain understanding about a problem domain and to predict the consequences of intervention. Three, because the model has both a causal and probabilistic semantics, it is an ideal representation for combining prior knowledge (which often comes in causa…
Prior work it builds on
-
Variational Bayes on manifoldsMinh-Ngoc Tran, D. Nguyen, Duy Nguyen · 2019 · Statistics and computing · cited 26 times · Not summarized yet
Abstract
Variational Bayes (VB) has become a widely-used tool for Bayesian inference in statistics and machine learning. Nonetheless, the development of the existing VB algorithms is so far generally restricted to the case where the variational parameter space is Euclidean, which hinders the potential broad application of VB methods. This paper extends the scope of VB to the case where the variational parameter space is a Riemannian manifold. We develop an efficient manifold-based VB algorithm that exploits both the geometric structure of the constraint parameter space and the information geometry of the manifold of VB approximating probability distributions. Our algorithm is provably convergent and …
-
Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate ShiftAlex J. Chan, Ahmed M. Alaa, Zhaozhi Qian et al. · 2020 · International Conference on Machine Learning · cited 46 times · Not summarized yet
Abstract
Modern neural networks have proven to be powerful function approximators, providing state-of-the-art performance in a multitude of applications. They however fall short in their ability to quantify confidence in their predictions - this is crucial in high-stakes applications that involve critical decision-making. Bayesian neural networks (BNNs) aim at solving this problem by placing a prior distribution over the network's parameters, thereby inducing a posterior distribution that encapsulates predictive uncertainty. While existing variants of BNNs based on Monte Carlo dropout produce reliable (albeit approximate) uncertainty estimates over in-distribution data, they tend to exhibit over-conf…
-
A practical tutorial on Variational BayesMinh-Ngoc Tran, Trong-Nghia Nguyen, Viet-Hung Dao · 2021 · cited 56 times · Not summarized yet
Abstract
This tutorial gives a quick introduction to Variational Bayes (VB), also called Variational Inference or Variational Approximation, from a practical point of view. The paper covers a range of commonly used VB methods and an attempt is made to keep the materials accessible to the wide community of data analysis practitioners. The aim is that the reader can quickly derive and implement their first VB algorithm for Bayesian inference with their data analysis problem. An end-user software package in Matlab together with the documentation can be found at https://vbayeslab.github.io/VBLabDocs/
-
Boosting Black-Box Variational Inference by Incorporating the Natural GradientFelix Trusheim, A. Condurache, A. Mertins · 2018 · International Conference on Pattern Recognition · cited 5 times · Not summarized yet
Abstract
In this paper we present a modification of the popular Black-Box Variational Inference (BBVI) approach which significantly improves the computational efficiency of the inference. We achieve this performance boost by replacing the standard gradient in the stochastic gradient ascent framework of BBVI with the natural gradient. Our experimental results (e.g. training of neutral networks) show that the proposed method outperforms the original BBVI algorithm on both synthetic and real data.
-
How to become a Bayesian in eight easy steps: An annotated reading listAlexander Etz, Q. F. Gronau, F. Dablander et al. · 2017 · Psychonomic Bulletin & Review · cited 110 times · Not summarized yet
Abstract
In this guide, we present a reading list to serve as a concise introduction to Bayesian data analysis. The introduction is geared toward reviewers, editors, and interested researchers who are new to Bayesian statistics. We provide commentary for eight recommended sources, which together cover the theoretical and practical cornerstones of Bayesian statistics in psychology and related sciences. The resources are presented in an incremental order, starting with theoretical foundations and moving on to applied issues. In addition, we outline an additional 32 articles and books that can be consulted to gain background knowledge about various theoretical specifics and Bayesian approaches to freque…
Papers that cite it
-
Robust Domain Generalisation with Causal Invariant Bayesian Neural NetworksGaël Gendron, Michael J. Witbrock, G. Dobbie · 2024 · arXiv.org · preprint, not peer reviewed · Not summarized yet
Abstract
Deep neural networks can obtain impressive performance on various tasks under the assumption that their training domain is identical to their target domain. Performance can drop dramatically when this assumption does not hold. One explanation for this discrepancy is the presence of spurious domain-specific correlations in the training data that the network exploits. Causal mechanisms, in the other hand, can be made invariant under distribution changes as they allow disentangling the factors of distribution underlying the data generation. Yet, learning causal mechanisms to improve out-of-distribution generalisation remains an under-explored area. We propose a Bayesian neural architecture that…
-
Efficient Training of Probabilistic Neural Networks for Survival AnalysisChristian Marius Lillelund, Martin Magris, C. Pedersen · 2024 · IEEE journal of biomedical and health informatics · cited 8 times · Not summarized yet
Abstract
Variational Inference (VI) is a commonly used technique for approximate Bayesian inference and uncertainty estimation in deep learning models, yet it comes at a computational cost, as it doubles the number of trainable parameters to represent uncertainty. This rapidly becomes challenging in high-dimensional settings and motivates the use of alternative techniques for inference, such as Monte Carlo Dropout (MCD) or Spectral-normalized Neural Gaussian Process (SNGP). However, such methods have seen little adoption in survival analysis, and VI remains the prevalent approach for training probabilistic neural networks. In this paper, we investigate how to train deep probabilistic survival models …
-
Bayesian Neural Networks With Physics-Informed Priors With Application to Boundary Layer VelocityLuca Menicali, David H. Richter, S. Castruccio · 2025 · IEEE Transactions on Neural Networks and Learning Systems · cited 7 times · Not summarized yet
Abstract
One of the most popular recent areas of machine learning predicates the use of neural networks (NNs) augmented by information about the underlying process in the form of partial differential equations (PDEs). These physics-informed NNs (PINNs) are obtained by penalizing the inference with a PDE and have been cast as a minimization problem currently lacking a formal approach to quantify the uncertainty. In this work, we propose a novel model-based framework that regards the PDE as a prior information of a deep Bayesian NN (BNN), physics-informed prior (PIP)-BNN. The prior is calibrated without data to resemble the PDE solution in the prior mean, while our degree of confidence in the PDE with …
-
Survey of Optimization Algorithms in Modern Neural NetworksR. Abdulkadirov, P. Lyakhov, N. Nagornov · 2023 · Mathematics · cited 133 times · Not summarized yet
Abstract
The main goal of machine learning is the creation of self-learning algorithms in many areas of human activity. It allows a replacement of a person with artificial intelligence in seeking to expand production. The theory of artificial neural networks, which have already replaced humans in many problems, remains the most well-utilized branch of machine learning. Thus, one must select appropriate neural network architectures, data processing, and advanced applied mathematics tools. A common challenge for these networks is achieving the highest accuracy in a short time. This problem is solved by modifying networks and improving data pre-processing, where accuracy increases along with training ti…
-
Uncertainty Estimation in Deep Bayesian Survival ModelsChristian Marius Lillelund, Martin Magris, C. Pedersen · 2023 · 2023 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI) · cited 8 times · Not summarized yet
Abstract
Bayesian methods can express uncertainty about their predictions, but have seen little adaptation in survival analysis using neural networks. Proper uncertainty estimation is important in high-risk domains, such as the healthcare or medical field, if machine learning methods are to be adopted for decision-making purposes, however, uncertainty estimation is a known shortcoming of neural networks. In this paper, we introduce the use of Bayesian inference techniques for survival analysis in neural networks that rely on the Cox proportional hazard assumption, for which we discuss a new flexible and effective architecture. We implement three architectures: a fully-deterministic neural network tha…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 5 November 2026.