# Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks

**Authors:** Sina Khajehabdollahi, Roxana Zeraati, Emmanouil Giannakakis, Tim Jakob Schäfer, Georg Martius, Anna Levina  
**arXiv:** [2309.12927](https://arxiv.org/abs/2309.12927)  
**Date:** 2023-09-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.12927v1/

## Key points

- The study focuses on understanding the mechanisms of solving memory-dependent tasks in recurrent neural networks (RNNs)
- RNNs are known for their ability to solve tasks with intricate temporal dependencies
- The specific contributions of individual neurons and recurrent interactions in solving such tasks are poorly understood
- Two types of memory-dependent tasks were used: $N$-parity and $N$-delayed match-to-sample
- Memory requirements controlled by parameter $N$, representing task complexity
- Recurrent weights and individual neuron timescales ($\tau$) were simultaneously optimized during training
- RNNs developed longer timescales as memory requirements increased (higher values of $N$)
- Two distinct curricula were used: single-head learning and multi-head learning
- Single-head networks increased individual neuron timescales with increasing $N$, but suffered from catastrophic forgetting
- Multi-head networks kept $\tau$ constant and developed longer timescales through recurrent connectivity, improving stability and generalization to new tasks
- Applying this curriculum significantly improved training GRUs and LSTMs for large-$N$ tasks
- Adapting timescales to task requirements through recurrent interactions allows RNNs to learn more complex objectives and improves performance

## Summary

The study focuses on understanding the mechanisms that contribute to solving memory-dependent tasks in recurrent neural networks (RNNs). RNNs are known for their ability to solve tasks with intricate temporal dependencies, but the specific contributions of individual neurons and recurrent interactions among them in solving such tasks remain poorly understood. To investigate this, the researchers trained RNNs to solve two types of memory-dependent tasks: $N$-parity and $N$-delayed match-to-sample. The memory requirements of these tasks were controlled by a parameter $N$, which represents the complexity of the task. The researchers simultaneously optimized both the recurrent weights and the individual neuron timescales ($\tau$) during training. The results showed that as the memory requirements increased (higher values of $N$), RNNs developed longer timescales. However, the specific mechanisms used by RNNs varied depending on the learning objective. Two distinct curricula were used to define learning objectives: sequential learning of a single-$N$ (single-head) or simultaneous learning of multiple $N$s (multi-head). Single-head networks increased their individual neuron timescales ($\tau$) with increasing $N$. These networks were able to solve tasks for large values of $N$, but they suffered from catastrophic forgetting, which refers to a loss of previously learned information when new information is learned. On the other hand, multi-head networks, which were explicitly required to hold multiple concurrent memories, kept their $\tau$ constant and developed longer timescales through recurrent connectivity. Furthermore, it was found that this curriculum improved training speed and network stability to ablations and perturbations. It also allowed RNNs to generalize better to tasks beyond their training regime. Additionally, applying this curriculum significantly improved training Gated Recurrent Units (GRUs) and Long Short-Term Memory (LSTM) models for large-$N$ tasks. Overall, these findings suggest that adapting timescales to task requirements through recurrent interactions allows RNNs to learn more complex objectives and improves their performance. The study provides insights into the mechanisms underlying memory-dependent tasks in RNNs and highlights the importance of curriculum design for effectively training these networks.