# Nonparametric Bayesian Structure Adaptation for Continual Learning

**Authors:** Abhishek Kumar, Sunabha Chatterjee, Piyush Rai  
**arXiv:** [1912.03624](https://arxiv.org/abs/1912.03624)  
**Date:** 2019-12-08  
**Summary page:** https://www.summarizepaper.com/en/paper/1912.03624v1/

## Key points

- Continual Learning is a learning paradigm for training machine learning models with sequential or streaming tasks
- Recent advances in continual learning with neural networks focus on variational Bayes based regularization and learning the structure of deep networks
- The authors propose a principled non-parametric Bayesian approach for learning the structure of feed-forward neural networks in continual learning
- Their model allows the number of nodes in each hidden layer to automatically grow with each new task, leading to better performance on benchmark datasets
- Inter-task transfer occurs through overlapping sparse subsets of weights learned by different tasks, enabling effective knowledge transfer and enhancing performance and efficiency
- The proposed model is validated on benchmark datasets and demonstrates comparable or superior performance compared to existing approaches
- The model can adaptively infer the evolving network structure in continual learning setting

## Summary

Continual Learning is a learning paradigm in which machine learning models are trained with sequential or streaming tasks. Recent advances in continual learning with neural networks have focused on two notable directions: (i) variational Bayes based regularization by learning priors from previous tasks, and (ii) learning the structure of deep networks to adapt to new tasks. However, these approaches have been orthogonal and have their own limitations. In this paper, the authors propose a principled non-parametric Bayesian approach for learning the structure of feed-forward neural networks in the context of continual learning. Their model addresses the shortcomings of both existing approaches. One key feature of their model is that it allows the number of nodes in each hidden layer to automatically grow with the introduction of each new task. This adaptive growth enables better performance on benchmark datasets compared to state-of-the-art approaches. Additionally, inter-task transfer occurs through the overlapping of different sparse subsets of weights learned by different tasks. This means that knowledge acquired from previous tasks can be effectively transferred to new tasks, enhancing overall performance and efficiency. The authors validate their proposed model on benchmark datasets and demonstrate its comparable or superior performance compared to existing approaches. Moreover, their model is able to adaptively infer the evolving network structure in the continual learning setting. Overall, this paper presents a novel approach for addressing the challenges in continual learning with neural networks by combining variational Bayes based regularization and adaptive network structure learning. The results highlight the effectiveness of their model in achieving improved performance while accommodating evolving task requirements.