# Is Mamba Capable of In-Context Learning?

**Authors:** Riccardo Grazzi, Julien Siems, Simon Schrodi, Thomas Brox, Frank Hutter  
**arXiv:** [2402.03170](https://arxiv.org/abs/2402.03170)  
**Date:** 2024-02-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.03170v1/

## Key points

- Mamba exhibits similar in-context learning (ICL) capabilities as transformer models
- Evaluations on various tasks show that Mamba performs at the same level as transformer models for ICL across both task categories
- Mamba optimizes its internal representations incrementally to solve ICL problems, similar to transformers
- Mamba can be an efficient alternative to transformers for ICL tasks involving longer input sequences
- Transformer models achieve ICL through pre-training without explicit training or fine-tuning, which has generated significant academic interest
- Recent studies have contributed to understanding how transformers implement and learn variants of in-context gradient descent during pre-training
- Mamba matches the performance of transformer models for ICL across both simple function approximation and complex natural language processing problems
- Both Mamba and transformers optimize their internal representations incrementally, suggesting a shared mechanism for effective in-context learning
- Further investigations are required to fully understand the extent of Mamba's ICL capabilities and its potential advantages over transformers

## Summary

, , , , 

This study presents empirical evidence that the selective structured state space model, Mamba, exhibits similar in-context learning (ICL) capabilities as transformer models. The authors conducted evaluations on various tasks, including simple function approximation and complex natural language processing problems. The results demonstrate that Mamba performs at the same level as transformer models for ICL across both task categories. Further analysis reveals that Mamba, like transformers, optimizes its internal representations incrementally to solve ICL problems. This suggests that Mamba can be an efficient alternative to transformers for ICL tasks involving longer input sequences. The introduction highlights recent advancements in large-scale neural language modeling and emphasizes the ICL capabilities of transformer models. These models are able to infer how to perform tasks solely from input examples after self-supervised pre-training without explicit training or fine-tuning. This departure from traditional machine learning approaches has generated significant academic interest. The authors note that while meta-learning approaches rely on explicit training on a distribution of tasks and specific inductive biases, transformer models achieve ICL through pre-training without such requirements. Recent studies have contributed to understanding how transformers implement and learn variants of in-context gradient descent during pre-training. To evaluate Mamba's ICL capabilities, the authors conducted experiments on tasks involving simple function approximation and more complex natural language processing problems. The results demonstrate that Mamba matches the performance of transformer models for ICL across both task categories. Further analysis reveals similarities between Mamba and transformers in terms of their approach to solving ICL problems. Both models optimize their internal representations incrementally, suggesting a shared mechanism for achieving effective in-context learning. Overall, this work provides empirical evidence supporting the notion that Mamba can be an efficient alternative to transformers for ICL tasks involving longer input sequences. Further investigations are required to fully understand the extent of Mamba's ICL capabilities and its potential advantages over transformers. In conclusion, this study contributes to the understanding of selective structured state space models and their ability to perform in-context learning. The findings suggest that Mamba can be a viable option for tasks requiring ICL, particularly when dealing with longer input sequences.