# An Introduction to Lifelong Supervised Learning

**Authors:** Shagun Sodhani, Mojtaba Faramarzi, Sanket Vaibhav Mehta, Pranshu Malviya, Mohamed Abdelsalam, Janarthanan Janarthanan, Sarath Chandar  
**arXiv:** [2207.04354](https://arxiv.org/abs/2207.04354)  
**Date:** 2022-07-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2207.04354v2/

## Key points

- Comprehensive overview of lifelong learning
- Divided into seven chapters, each focusing on different aspects
- Chapter 2: High-level discussion on lifelong learning systems
- Different scenarios and approaches
- Ideal characteristics of a system
- Relationship with other paradigms
- Common metrics for evaluation
- Chapter 3: Regularization-based methods discussed
- Importance-based regularization and distillation-based regularization techniques
- Chapter 4: Memory-based approaches in lifelong learning explored
- Unified view of episodic memory for continual learning
- Strategies for using memory at test time, generative replay techniques, and summary of methods
- Chapter 5: Architecture-based approaches for training lifelong learning systems covered
- Modular networks and parameter isolation systems discussed
- Overview of different architecture families and their instantiations
- Chapter 6: Evaluation benchmarks and metrics for assessing performance discussed
- Vision benchmarks and NLP benchmarks covered
- Summary of commonly used evaluation criteria in this domain
- Chapter 7: Future challenges and research directions in the field addressed

## Summary

This comprehensive primer on lifelong learning provides a thorough overview of the various facets of this field. It is divided into seven chapters, each focusing on a different aspect of lifelong learning. Chapter 2 offers a high-level discussion on lifelong learning systems. It covers different scenarios and approaches to lifelong learning, outlines the ideal characteristics of a system in this field, explores its relationship with other paradigms, and explains common metrics used for evaluation. This chapter is particularly beneficial for readers new to the concept. Subsequent chapters delve into specific approaches for training lifelong learning systems. Chapter 3 focuses on regularization-based methods that do not rely on data from previous tasks. It discusses various techniques such as importance-based regularization and distillation-based regularization. Chapter 4 explores memory-based approaches in lifelong learning. It presents a unified view of episodic memory for continual learning and discusses strategies for using memory at test time. Additionally, it covers generative replay techniques and summarizes memory-based methods. In Chapter 5, the document delves into architecture-based approaches for training lifelong learning systems. It discusses modular networks and parameter isolation systems while providing an overview of different architecture families and their instantiations. Chapter 6 shifts focus to evaluation benchmarks and metrics for assessing the performance of lifelong learning systems. It covers vision benchmarks and NLP benchmarks while summarizing commonly used evaluation criteria in this domain. The document wraps up with Chapter 7 which addresses future challenges and research directions in the field of lifelong supervised learning. Overall, this primer serves as a valuable resource for individuals interested in understanding the complexities and advancements in lifelong learning systems.