# xLSTM: Extended Long Short-Term Memory

**Authors:** Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, Sepp Hochreiter  
**arXiv:** [2405.04517](https://arxiv.org/abs/2405.04517)  
**Date:** 2024-05-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.04517v1/

## Key points

- Long Short-Term Memory (LSTM) models introduced in the 1990s with concepts of constant error carousel and gating
- LSTM instrumental in deep learning success stories, particularly in Large Language Models (LLMs)
- Emergence of Transformer technology with parallelizable self-attention capabilities surpassing LSTMs in scalability
- Introduction of Extended Long Short-Term Memory (xLSTM) model to address effectiveness of scaled LSTMs incorporating modern techniques from LLMs
- xLSTM incorporates exponential gating with normalization and stabilization techniques, sLSTM with scalar memory, and mLSTM fully parallelizable with matrix memory
- Performance evaluation of xLSTM through experiments on formal languages, Multi-Query Associative Recall tasks, Long Range Arena sequences, and SlimPajama dataset containing 15B tokens
- Ablation studies on xLSTM for better understanding of performance
- Comparison against other top-performing methods using 300B tokens from SlimPajama dataset for training in comprehensive language modeling experiment
- Evaluation criteria include extrapolation to longer contexts, validation perplexity scores, performance on downstream tasks, assessment across multiple text domains using PALOMA benchmark dataset, and scaling behavior analysis

## Summary

In the 1990s, Long Short-Term Memory (LSTM) models were introduced with the concepts of constant error carousel and gating. These have since been instrumental in various deep learning success stories, particularly in the development of Large Language Models (LLMs). However, the emergence of Transformer technology with parallelizable self-attention capabilities has ushered in a new era, surpassing LSTMs in terms of scalability. This led to a fundamental question: How effective can LSTMs be in language modeling when scaled to billions of parameters while incorporating modern techniques from LLMs and addressing known limitations? To address this question, a new approach called Extended Long Short-Term Memory (xLSTM) was proposed. The xLSTM model incorporates exponential gating with normalization and stabilization techniques, as well as modifications to the LSTM memory structure. These modifications include introducing sLSTM with scalar memory and update mechanisms, alongside mLSTM which is fully parallelizable with matrix memory and covariance update rules. Through a series of experiments detailed in Section 4, xLSTM's performance was evaluated and compared against existing methods in language modeling tasks. Synthetic tasks were used to test xLSTM's effectiveness on formal languages and Multi-Query Associative Recall tasks. Additionally, xLSTM's ability to process long sequences was assessed in the Long Range Arena. Furthermore, extensive experiments were conducted using the SlimPajama dataset containing 15B tokens for training and evaluation. Ablation studies were performed on xLSTM to understand its performance better. Scaling behavior comparisons similar to previous studies by Kaplan et al. (2020) and Brown et al. (2020) were also carried out. In a more comprehensive language modeling experiment using 300B tokens from SlimPajama dataset for training,xLSTM was compared against other top-performing methods. Evaluation criteria included extrapolation to longer contexts, validation perplexity scores, performance on downstream tasks such as those outlined by Sutawika et al., 2024), assessment across multiple text domains using the PALOMA benchmark dataset by Magnusson et al., 2023), and scaling behavior analysis with increased training data. Overall, through these experiments and evaluations, xLSTM demonstrated favorable performance compared to state-of-the-art Transformers and State Space Models both in terms of efficiency and scalability for language modeling tasks.