Bayesian Learning for Neural Networks: an algorithmic survey
Auteurs : Martin Magris, Alexandros Iosifidis
Résumé scientifique
The last decade witnessed a growing interest in Bayesian learning. Yet, the technicality of the topic and the multitude of ingredients involved therein, besides the complexity of turning theory into practical implementations, limit the use of the Bayesian learning paradigm, preventing its widespread adoption across different fields and applications. This self-contained survey engages and introduces readers to the principles and algorithms of Bayesian Learning for Neural Networks. It provides an introduction to the topic from an accessible, practical-algorithmic perspective. Upon providing a general introduction to Bayesian Neural Networks, we discuss and present both standard and recent approaches for Bayesian inference, with an emphasis on solutions relying on Variational Inference and the use of Natural gradients. We also discuss the use of manifold optimization as a state-of-the-art approach to Bayesian learning. We examine the characteristic properties of all the discussed methods, and provide pseudo-codes for their implementation, paying attention to practical aspects, such as the computation of the gradients
Les articles autour de lui, par date et par proximité
Plus haut, c’est plus proche de cet article ; à gauche, c’est plus ancien. Les traits relient les articles proches entre eux : les groupes de traits sont des sous-thèmes. Sélectionnez un point pour voir pourquoi il est là. Retour au résumé de cet article
Les articles autour de celui-ci
Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.
Les articles les plus proches
-
A Primer on Bayesian Neural Networks: Review and DebatesJulyan Arbel, Konstantinos Pitas, Mariia Vladimirova et al. · 2023 · Machine Learning · preprint, non relu par les pairs · cité 60 fois · Résumé IA disponible
Résumé scientifique
Neural networks have achieved remarkable performance across various problem domains, but their widespread applicability is hindered by inherent limitations such as overconfidence in predictions, lack of interpretability, and vulnerability to adversarial attacks. To address these challenges, Bayesian neural networks (BNNs) have emerged as a compelling extension of conventional neural networks, integrating uncertainty estimation into their predictive capabilities. This comprehensive primer presents a systematic introduction to the fundamental concepts of neural networks and Bayesian inference, elucidating their synergistic integration for the development of BNNs. The target audience comprises …
-
Hands-On Bayesian Neural Networks—A Tutorial for Deep Learning UsersLaurent Valentin Jospin, Hamid Laga, F. Boussaid et al. · 2020 · IEEE Computational Intelligence Magazine · cité 977 fois · Pas encore résumé
Résumé scientifique
Modern deep learning methods constitute incredibly powerful tools to tackle a myriad of challenging problems. However, since deep learning methods operate as black boxes, the uncertainty associated with their predictions is often challenging to quantify. Bayesian statistics offer a formalism to understand and quantify the uncertainty associated with deep neural network predictions. This tutorial provides deep learning practitioners with an overview of the relevant literature and a complete toolset to design, implement, train, use and evaluate Bayesian neural networks, i.e., stochastic artificial neural networks trained using Bayesian methods.
-
Practical Deep Learning with Bayesian PrinciplesKazuki Osawa, S. Swaroop, Mohammad Emtiyaz Khan et al. · 2019 · Neural Information Processing Systems · cité 288 fois · Pas encore résumé
Résumé scientifique
Bayesian methods promise to fix many shortcomings of deep learning, but they are impractical and rarely match the performance of standard methods, let alone improve them. In this paper, we demonstrate practical training of deep networks with natural-gradient variational inference. By applying techniques such as batch normalisation, data augmentation, and distributed training, we achieve similar performance in about the same number of epochs as the Adam optimiser, even on large datasets such as ImageNet. Importantly, the benefits of Bayesian principles are preserved: predictive probabilities are well-calibrated, uncertainties on out-of-distribution data are improved, and continual-learning pe…
-
DE-BNN: An evolutionary approach to Bayesian neural network posterior samplingWesley Forbes, Min Long · 2026 · Neurocomputing · Pas encore résumé
-
Priors in Bayesian Deep Learning: A ReviewVincent Fortuin · 2021 · Machine Learning · cité 201 fois · Résumé IA disponible
Résumé scientifique
While the choice of prior is one of the most critical parts of the Bayesian inference workflow, recent Bayesian deep learning models have often fallen back on vague priors, such as standard Gaussians. In this review, we highlight the importance of prior choices for Bayesian deep learning and present an overview of different priors that have been proposed for (deep) Gaussian processes, variational autoencoders, and Bayesian neural networks. We also outline different methods of learning priors for these models from data. We hope to motivate practitioners in Bayesian deep learning to think more carefully about the prior specification for their models and to provide them with some inspiration in…
-
Bayesian Transfer LearningPiotr M. Suder, Jason Xu, David B. Dunson · 2023 · Machine Learning · cité 17 fois · Résumé IA disponible
Résumé scientifique
Transfer learning is a burgeoning concept in statistical machine learning that seeks to improve inference and/or predictive accuracy on a domain of interest by leveraging data from related domains. While the term "transfer learning" has garnered much recent interest, its foundational principles have existed for years under various guises. Prior literature reviews in computer science and electrical engineering have sought to bring these ideas into focus, primarily surveying general methodologies and works from these disciplines. This article highlights Bayesian approaches to transfer learning, which have received relatively limited attention despite their innate compatibility with the notion …
-
Variational Bayesian Learning TheoryShinichi Nakajima, Kazuho Watanabe, Masashi Sugiyama · 2019 · cité 32 fois · Pas encore résumé
Résumé scientifique
Variational Bayesian learning is one of the most popular methods in machine learning. Designed for researchers and graduate students in machine learning, this book summarizes recent developments in the non-asymptotic and asymptotic theory of variational Bayesian learning and suggests how this theory can be applied in practice. The authors begin by developing a basic framework with a focus on conjugacy, which enables the reader to derive tractable algorithms. Next, it summarizes non-asymptotic theory, which, although limited in application to bilinear models, precisely describes the behavior of the variational Bayesian solution and reveals its sparsity inducing mechanism. Finally, the text su…
-
A Tutorial on Learning with Bayesian NetworksD. Heckerman · 1999 · Innovations in Bayesian Networks · cité 3 655 fois · Pas encore résumé
Résumé scientifique
A Bayesian network is a graphical model that encodes probabilistic relationships among variables of interest. When used in conjunction with statistical techniques, the graphical model has several advantages for data analysis. One, because the model encodes dependencies among all variables, it readily handles situations where some data entries are missing. Two, a Bayesian network can be used to learn causal relationships, and hence can be used to gain understanding about a problem domain and to predict the consequences of intervention. Three, because the model has both a causal and probabilistic semantics, it is an ideal representation for combining prior knowledge (which often comes in causa…
Les travaux antérieurs sur lesquels il s’appuie
-
Variational Bayes on manifoldsMinh-Ngoc Tran, D. Nguyen, Duy Nguyen · 2019 · Statistics and computing · cité 26 fois · Pas encore résumé
Résumé scientifique
Variational Bayes (VB) has become a widely-used tool for Bayesian inference in statistics and machine learning. Nonetheless, the development of the existing VB algorithms is so far generally restricted to the case where the variational parameter space is Euclidean, which hinders the potential broad application of VB methods. This paper extends the scope of VB to the case where the variational parameter space is a Riemannian manifold. We develop an efficient manifold-based VB algorithm that exploits both the geometric structure of the constraint parameter space and the information geometry of the manifold of VB approximating probability distributions. Our algorithm is provably convergent and …
-
Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate ShiftAlex J. Chan, Ahmed M. Alaa, Zhaozhi Qian et al. · 2020 · International Conference on Machine Learning · cité 46 fois · Pas encore résumé
Résumé scientifique
Modern neural networks have proven to be powerful function approximators, providing state-of-the-art performance in a multitude of applications. They however fall short in their ability to quantify confidence in their predictions - this is crucial in high-stakes applications that involve critical decision-making. Bayesian neural networks (BNNs) aim at solving this problem by placing a prior distribution over the network's parameters, thereby inducing a posterior distribution that encapsulates predictive uncertainty. While existing variants of BNNs based on Monte Carlo dropout produce reliable (albeit approximate) uncertainty estimates over in-distribution data, they tend to exhibit over-conf…
-
A practical tutorial on Variational BayesMinh-Ngoc Tran, Trong-Nghia Nguyen, Viet-Hung Dao · 2021 · cité 56 fois · Pas encore résumé
Résumé scientifique
This tutorial gives a quick introduction to Variational Bayes (VB), also called Variational Inference or Variational Approximation, from a practical point of view. The paper covers a range of commonly used VB methods and an attempt is made to keep the materials accessible to the wide community of data analysis practitioners. The aim is that the reader can quickly derive and implement their first VB algorithm for Bayesian inference with their data analysis problem. An end-user software package in Matlab together with the documentation can be found at https://vbayeslab.github.io/VBLabDocs/
-
Boosting Black-Box Variational Inference by Incorporating the Natural GradientFelix Trusheim, A. Condurache, A. Mertins · 2018 · International Conference on Pattern Recognition · cité 5 fois · Pas encore résumé
Résumé scientifique
In this paper we present a modification of the popular Black-Box Variational Inference (BBVI) approach which significantly improves the computational efficiency of the inference. We achieve this performance boost by replacing the standard gradient in the stochastic gradient ascent framework of BBVI with the natural gradient. Our experimental results (e.g. training of neutral networks) show that the proposed method outperforms the original BBVI algorithm on both synthetic and real data.
-
How to become a Bayesian in eight easy steps: An annotated reading listAlexander Etz, Q. F. Gronau, F. Dablander et al. · 2017 · Psychonomic Bulletin & Review · cité 110 fois · Pas encore résumé
Résumé scientifique
In this guide, we present a reading list to serve as a concise introduction to Bayesian data analysis. The introduction is geared toward reviewers, editors, and interested researchers who are new to Bayesian statistics. We provide commentary for eight recommended sources, which together cover the theoretical and practical cornerstones of Bayesian statistics in psychology and related sciences. The resources are presented in an incremental order, starting with theoretical foundations and moving on to applied issues. In addition, we outline an additional 32 articles and books that can be consulted to gain background knowledge about various theoretical specifics and Bayesian approaches to freque…
Les articles qui le citent
-
Robust Domain Generalisation with Causal Invariant Bayesian Neural NetworksGaël Gendron, Michael J. Witbrock, G. Dobbie · 2024 · arXiv.org · preprint, non relu par les pairs · Pas encore résumé
Résumé scientifique
Deep neural networks can obtain impressive performance on various tasks under the assumption that their training domain is identical to their target domain. Performance can drop dramatically when this assumption does not hold. One explanation for this discrepancy is the presence of spurious domain-specific correlations in the training data that the network exploits. Causal mechanisms, in the other hand, can be made invariant under distribution changes as they allow disentangling the factors of distribution underlying the data generation. Yet, learning causal mechanisms to improve out-of-distribution generalisation remains an under-explored area. We propose a Bayesian neural architecture that…
-
Efficient Training of Probabilistic Neural Networks for Survival AnalysisChristian Marius Lillelund, Martin Magris, C. Pedersen · 2024 · IEEE journal of biomedical and health informatics · cité 8 fois · Pas encore résumé
Résumé scientifique
Variational Inference (VI) is a commonly used technique for approximate Bayesian inference and uncertainty estimation in deep learning models, yet it comes at a computational cost, as it doubles the number of trainable parameters to represent uncertainty. This rapidly becomes challenging in high-dimensional settings and motivates the use of alternative techniques for inference, such as Monte Carlo Dropout (MCD) or Spectral-normalized Neural Gaussian Process (SNGP). However, such methods have seen little adoption in survival analysis, and VI remains the prevalent approach for training probabilistic neural networks. In this paper, we investigate how to train deep probabilistic survival models …
-
Bayesian Neural Networks With Physics-Informed Priors With Application to Boundary Layer VelocityLuca Menicali, David H. Richter, S. Castruccio · 2025 · IEEE Transactions on Neural Networks and Learning Systems · cité 7 fois · Pas encore résumé
Résumé scientifique
One of the most popular recent areas of machine learning predicates the use of neural networks (NNs) augmented by information about the underlying process in the form of partial differential equations (PDEs). These physics-informed NNs (PINNs) are obtained by penalizing the inference with a PDE and have been cast as a minimization problem currently lacking a formal approach to quantify the uncertainty. In this work, we propose a novel model-based framework that regards the PDE as a prior information of a deep Bayesian NN (BNN), physics-informed prior (PIP)-BNN. The prior is calibrated without data to resemble the PDE solution in the prior mean, while our degree of confidence in the PDE with …
-
Survey of Optimization Algorithms in Modern Neural NetworksR. Abdulkadirov, P. Lyakhov, N. Nagornov · 2023 · Mathematics · cité 133 fois · Pas encore résumé
Résumé scientifique
The main goal of machine learning is the creation of self-learning algorithms in many areas of human activity. It allows a replacement of a person with artificial intelligence in seeking to expand production. The theory of artificial neural networks, which have already replaced humans in many problems, remains the most well-utilized branch of machine learning. Thus, one must select appropriate neural network architectures, data processing, and advanced applied mathematics tools. A common challenge for these networks is achieving the highest accuracy in a short time. This problem is solved by modifying networks and improving data pre-processing, where accuracy increases along with training ti…
-
Uncertainty Estimation in Deep Bayesian Survival ModelsChristian Marius Lillelund, Martin Magris, C. Pedersen · 2023 · 2023 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI) · cité 8 fois · Pas encore résumé
Résumé scientifique
Bayesian methods can express uncertainty about their predictions, but have seen little adaptation in survival analysis using neural networks. Proper uncertainty estimation is important in high-risk domains, such as the healthcare or medical field, if machine learning methods are to be adopted for decision-making purposes, however, uncertainty estimation is a known shortcoming of neural networks. In this paper, we introduce the use of Bayesian inference techniques for survival analysis in neural networks that rely on the Cox proportional hazard assumption, for which we discuss a new flexible and effective architecture. We implement three architectures: a fully-deterministic neural network tha…
Aller plus loin
Comment ces articles sont-ils trouvés ?
Nous comparons le titre et le résumé de cet article avec ceux de tous les articles résumés sur ce site, et avec les articles que Semantic Scholar recommande, ceux qu’il cite et ceux qui le citent (pour un article très cité, ses citations les plus récentes et, depuis OpenAlex, les plus citées). Parmi les travaux antérieurs et les suites, les articles qui s’utilisent en profondeur, une méthode qu’il utilise et les suites très citées remontent un peu. Les niveaux (très proche, proche, apparenté) viennent de la similarité des textes, pas d’un jugement de qualité ; * signale un article sans résumé, comparé par son titre. La phrase sous un article est écrite par une IA à partir des deux résumés : vérifiez-la dans les articles. Un article jugé hors sujet par trois lecteurs quitte la liste.
Mis à jour le 5 novembre 2026.