# Were RNNs All We Needed?

**Authors:** Leo Feng, Frederick Tung, Mohamed Osama Ahmed, Yoshua Bengio, Hossein Hajimirsadegh  
**arXiv:** [2410.01201](https://arxiv.org/abs/2410.01201)  
**Date:** 2024-10-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.01201v1/

## Key points

- Authors address limitations of Transformers in handling long sequences
- Introduction of novel architectures S4, Mamba, and Aaren achieving comparable performance to Transformers
- Focus on traditional RNNs like LSTMs and GRUs from over a decade ago
- Simplification of LSTMs and GRUs into minimal versions with fewer parameters for parallel training
- Revisiting LSTMs and GRUs with modern perspective inspired by recent advancements in state-space models and attention-based models
- Background information on RNNs suitability for capturing temporal dependencies in sequential tasks
- Experimental results showing comparable performance of minimal versions of LSTMs and GRUs to advanced models like Mamba and Transformers
- Emphasis on efficiency gains through simplifying traditional RNN architectures while maintaining competitive performance

## Summary

In their paper titled "Attention is All You Need," the authors address the limitations of Transformers in handling long sequences and the renewed interest in parallelizable recurrent sequence models. They introduce novel architectures such as S4, Mamba, and Aaren that achieve comparable performance to Transformers. The focus then shifts to traditional recurrent neural networks (RNNs) from over a decade ago, specifically Long Short-Term Memory (LSTM) networks introduced by Hochreiter & Schmidhuber in 1997 and Gated Recurrent Units (GRUs) by Cho et al. in 2014. The authors highlight that LSTMs and GRUs were hindered by backpropagation through time (BPTT), but by removing hidden state dependencies from their gates, they can be trained efficiently in parallel. They further simplify these models into minimal versions (minLSTMs and minGRUs) with fewer parameters that are fully parallelizable during training, resulting in a significant speedup for sequences of length 512. Drawing inspiration from recent advancements in state-space models like Mamba and attention-based models proposed by Peng et al. and Feng et al., the authors revisit LSTMs and GRUs with a modern perspective. By removing constraints on output range and ensuring time-independent scaling of outputs, they create stripped-down versions of these decade-old RNNs that match the empirical performance of recent sequence models. The paper also provides background information on RNNs, highlighting their suitability for capturing temporal dependencies in sequential tasks such as time series analysis and natural language processing. It discusses the limitations of vanilla RNNs due to vanishing gradients and introduces LSTM networks as an effective solution for learning long-term dependencies. Experimental results presented in tables show that minimal versions of LSTMs and GRUs perform comparably to advanced models like Mamba and Transformers on tasks such as the selective copy task and language modeling. In conclusion, the study emphasizes the efficiency gains achieved through simplifying traditional RNN architectures while maintaining competitive performance with state-of-the-art sequence models.