# Hungry Hungry Hippos: Towards Language Modeling with State Space Models

**Authors:** Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, Christopher Ré  
**arXiv:** [2212.14052](https://arxiv.org/abs/2212.14052)  
**Date:** 2022-12-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2212.14052v3/

## Key points

- State space models (SSMs) underperform attention-based models like Transformers in language modeling tasks
- SSMs are slower than Transformers due to inefficient hardware utilization
- The authors propose a new SSM layer called H3 to address the limitations of existing SSMs, achieving comparable performance to attention on synthetic languages and coming close to Transformers on OpenWebText dataset
- A hybrid model combining H3-attention outperforms Transformers on OpenWebText dataset by 1.0 perplexity per token (PPL)
- FlashConv technique improves efficiency for training SSMs on modern hardware, achieving a 2x speedup on long-range arena benchmark and enabling faster text generation than Transformers
- Hybrid H3-attention language models scaled up to 2.7B parameters achieve lower perplexity than Transformers and outperform them in zero- and few-shot learning on SuperGLUE benchmark

## Summary

The paper titled "Hungry Hungry Hippos: Towards Language Modeling with State Space Models" explores the performance of state space models (SSMs) in language modeling compared to attention-based models like Transformers. While SSMs have shown excellent sequence modeling performance in some domains, they underperform attention in language modeling tasks. Additionally, despite their linear scaling in sequence length, SSMs are slower than Transformers due to inefficient hardware utilization. To bridge the gap between SSMs and attention in language modeling, the authors conduct experiments using synthetic language modeling tasks. They identify two key challenges that existing SSMs struggle with: recalling earlier tokens in the sequence and comparing tokens across the sequence. To address these limitations, they propose a new SSM layer called H3, specifically designed for these capabilities. The H3 layer achieves comparable performance to attention on synthetic languages and comes within 0.4 perplexity per token (PPL) of Transformers on OpenWebText dataset. Furthermore, the authors introduce a hybrid model combining a 125M-parameter H3-attention model with two attention layers. Surprisingly, this hybrid model outperforms Transformers by 1.0 PPL on OpenWebText dataset. To improve the efficiency of training SSMs on modern hardware, the authors propose FlashConv. This technique utilizes a fused block Fast Fourier Transform (FFT) algorithm to enhance efficiency for sequences up to 8K length. It also introduces a novel state passing algorithm that leverages the recurrent properties of SSMs to scale to longer sequences. FlashConv achieves a 2x speedup on the long-range arena benchmark and enables hybrid language models to generate text 2.4x faster than Transformers. Using FlashConv, the authors scale up hybrid H3-attention language models up to 2.7B parameters on the Pile dataset and achieve promising results. These models achieve lower perplexity than Transformers and outperform Transformers in zero- and few-shot learning on a majority of tasks in the SuperGLUE benchmark. Overall, this paper provides insights into the expressivity gap between SSMs and attention in language modeling and proposes solutions to improve their performance.