# Efficiently Modeling Long Sequences with Structured State Spaces

**Authors:** Albert Gu, Karan Goel, Christopher Ré  
**arXiv:** [2111.00396](https://arxiv.org/abs/2111.00396)  
**Date:** 2021-10-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2111.00396v1/

## Key points

- Authors address the challenge of designing a single principled model for sequence modeling that can handle long-range dependencies
- Introduce the Structured State Space (S4) sequence model as a novel approach to overcome limitations of traditional models like RNNs, CNNs, and Transformers
- S4 model achieves stable diagonalization of the state matrix by conditioning \( A \) with a low-rank correction, simplifying computation to that of a Cauchy kernel
- Empirical results show effectiveness on various benchmarks including sequential CIFAR-10 and image and language modeling tasks
- Outperforms existing approaches on every task in the Long Range Arena benchmark
- Despite superior performance, S4 remains computationally efficient compared to competitors

## Summary

In their paper titled "Efficiently Modeling Long Sequences with Structured State Spaces," authors Albert Gu, Karan Goel, and Christopher Ré address the challenge of designing a single principled model for sequence modeling that can effectively handle long-range dependencies across various modalities and tasks. The authors introduce the Structured State Space (S4) sequence model as a novel approach to address the limitations of traditional models such as RNNs, CNNs, and Transformers in handling sequences of 10,000 or more steps. By conditioning \( A \) with a low-rank correction, the S4 model achieves stable diagonalization of the state matrix and simplifies computation to that of a Cauchy kernel. Empirical results demonstrate its effectiveness on various benchmarks including sequential CIFAR-10 and image and language modeling tasks. Furthermore, S4 outperforms existing approaches on every task in the Long Range Arena benchmark. Despite its superior performance, S4 remains computationally efficient compared to its competitors. Overall, the Structured State Space (S4) sequence model presents a promising solution for efficiently modeling long sequences with strong empirical results across diverse benchmarks and tasks.