# Plastic Learning with Deep Fourier Features

**Authors:** Alex Lewandowski, Dale Schuurmans, Marlos C. Machado  
**arXiv:** [2410.20634](https://arxiv.org/abs/2410.20634)  
**Date:** 2024-10-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.20634v1/

## Key points

- Authors address the challenge of continual learning in deep neural networks
- Loss of plasticity is a key issue identified
- Theoretical results show that linear function approximation and deep linear networks do not suffer from loss of plasticity
- Proposal of deep Fourier features involving sine and cosine concatenation in every layer
- Deep Fourier features strike a balance between trainability and effectiveness in neural networks
- Networks with deep Fourier features exhibit high trainability throughout learning process
- Empirical results show significant improvements in continual learning performance when replacing ReLU activations with deep Fourier features
- Improvements observed across various continual learning scenarios on datasets like CIFAR10, CIFAR100, and tiny-ImageNet
- Deep Fourier features are effective in diminishing label noise settings
- Networks using deep Fourier features consistently achieve high test accuracy despite initial challenges with corrupted labels on early tasks

## Summary

In the paper "Plastic Learning with Deep Fourier Features," authors Alex Lewandowski, Dale Schuurmans, and Marlos C. Machado address the challenge of continual learning in deep neural networks. They specifically focus on the phenomenon known as loss of plasticity and identify underlying principles that lead to plastic algorithms. The authors provide theoretical results showing that linear function approximation and a special case of deep linear networks do not suffer from loss of plasticity. To overcome this challenge, the authors propose deep Fourier features which involve the concatenation of a sine and cosine in every layer of a neural network. This combination strikes a dynamic balance between trainability achieved through linearity and effectiveness obtained through nonlinearity in neural networks. Networks composed entirely of deep Fourier features exhibit high trainability throughout the learning process. Empirical results presented in the paper demonstrate significant improvements in continual learning performance when ReLU activations are replaced with deep Fourier features. These improvements are observed across various continual learning scenarios such as label noise, class incremental learning, and pixel permutations on popular supervised learning datasets including CIFAR10, CIFAR100, and tiny-ImageNet. Further experiments conducted by the authors highlight the benefits of deep Fourier features in diminishing label noise settings. Despite initial challenges with corrupted labels on early tasks, networks utilizing deep Fourier features consistently achieve high test accuracy on uncorrupted test sets as label noise diminishes over subsequent tasks. Overall, this study underscores the effectiveness of adaptive-linearity as an inductive bias for continual learning in deep neural networks. The incorporation of deep Fourier features offers a promising approach to enhancing trainability and generalization capabilities in evolving environments.