# MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts

**Authors:** Maciej Pióro, Kamil Ciebiera, Krystian Król, Jan Ludziejewski, Sebastian Jaszczur  
**arXiv:** [2401.04081](https://arxiv.org/abs/2401.04081)  
**Date:** 2024-01-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.04081v1/

## Key points

- State Space Models (SSMs) and Mixture of Experts (MoE) are challenging the dominance of Transformers in sequential modeling.
- The authors propose combining SSMs with MoE to unlock their potential for scaling.
- The authors showcase their approach on a recent SSM-based model called Mamba, which achieves remarkable performance similar to Transformers.
- They introduce their model, MoE-Mamba, which outperforms both Mamba and Transformer-MoE.
- MoE-Mamba achieves the same performance as Mamba but with 2.2 times fewer training steps.
- It preserves the inference performance gains of Mamba against Transformers.
- This research contributes to advancing sequential modeling techniques by opening up new possibilities for scaling SSMs using MoE.

## Summary

, , , , 

The field of sequential modeling has seen significant advancements with the emergence of State Space Models (SSMs) as strong contenders challenging the dominance of Transformers. At the same time, Mixture of Experts (MoE) has greatly improved Transformer-based Language Models (LLMs), including recent state-of-the-art open-source models. In this paper, the authors propose combining SSMs with MoE to unlock their potential for scaling. The authors showcase their approach on a recent SSM-based model called Mamba, which achieves remarkable performance similar to Transformers. They introduce their model, MoE-Mamba, which outperforms both Mamba and Transformer-MoE. Notably, MoE-Mamba achieves the same performance as Mamba but with 2.2 times fewer training steps. Additionally, it preserves the inference performance gains of Mamba against Transformers. This research highlights the effectiveness of combining SSMs and MoE in achieving efficient selective state space models and contributes to advancing sequential modeling techniques by opening up new possibilities for scaling SSMs using MoE.