# Mamba: Linear-Time Sequence Modeling with Selective State Spaces

**Authors:** Albert Gu, Tri Dao  
**arXiv:** [2312.00752](https://arxiv.org/abs/2312.00752)  
**Date:** 2023-12-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.00752v1/

## Key points

- Mamba is a novel model that improves upon existing Transformer-based models by incorporating selective SSMs for content-based reasoning.
- Mamba achieves state-of-the-art performance across multiple modalities including language, audio, and genomics.
- The Mamba model allows for content-based reasoning and improved performance on various modalities such as language, audio, and genomics.
- The authors compare Mamba to the SaShiMi architecture and training protocols in the context of audio modelling and generation.
- They consider replacing the S4+MLP blocks in SaShiMi with Mamba blocks and provide experiment details in Appendix E.4.
- The evaluation of pretraining quality for audio waveform modality is conducted using the YouTubeMix dataset which consists of 4 hours of solo piano music sampled at a rate of 16000 Hz.
- Standard language modelling setup is followed for pretraining other modalities, evaluating the constant factor log(2) of the standard negative log-likelihood (NLL) loss for these modalities.
- For digit recognition, a dataset comprising 1-second clips sampled at 16000 Hz of digits "zero" through "nine" with highly variable characteristics is used.
- The authors' affiliations are provided as Albert Gu from Carnegie Mellon University's Machine Learning Department and Tri Dao from Princeton University's Department of Computer Science.

## Summary

This research introduces Mamba as a novel model that improves upon existing Transformer-based models by incorporating selective SSMs for content-based reasoning. It achieves state-of-the-art performance across multiple modalities including language, audio, and genomics. The Mamba model allows for content-based reasoning and improved performance on various modalities such as language, audio, and genomics. In the context of audio modelling and generation, the authors compare Mamba to the SaShiMi architecture and training protocols. They consider replacing the S4+MLP blocks in SaShiMi with Mamba blocks and provide experiment details in Appendix E.4. The evaluation of pretraining quality for audio waveform modality is conducted using the YouTubeMix dataset which consists of 4 hours of solo piano music sampled at a rate of 16000 Hz. Additionally, they follow standard language modelling setup for pretraining other modalities and evaluate the constant factor log(2) of the standard negative log-likelihood (NLL) loss for these modalities. For digit recognition they use a dataset comprising 1-second clips sampled at 16000 Hz of digits "zero" through "nine" with highly variable characteristics. The authors' affiliations are provided as Albert Gu from Carnegie Mellon University's Machine Learning Department and Tri Dao from Princeton University's Department of Computer Science.