# Flan-MoE: Scaling Instruction-Finetuned Language Models with Sparse Mixture of Experts

**Authors:** Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, Denny Zhou  
**arXiv:** [2305.14705](https://arxiv.org/abs/2305.14705)  
**Date:** 2023-05-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.14705v1/

## Key points

- Flan-MoE is introduced to address the demand for efficient and scalable language models
- Instruction-finetuning is crucial for enhancing MoE model performance
- Flan-MoE outperforms dense models in various experiment settings
- Flan-MoE-32B surpasses Flan-PaLM-62B across four benchmarks with one-third of the FLOPs
- Training data includes 1,836 finetuning tasks from a combination of four mixtures
- Evaluations conducted through zero-shot and few-shot assessments on held-out tasks
- Various benchmarks like MMLU, BBH, GSM8K, SVAMP/ASDIV, and StrategyQA are utilized for evaluation purposes

## Summary

In this paper, the authors introduce Flan-MoE, a set of  designed to address the increasing demand for efficient and scalable methods in the realm of language models. The study reveals that simply finetuning MoE models on task-specific datasets without instruction-finetuning often leads to inferior performance compared to dense models with similar computational complexity. However, through their experiments, the researchers demonstrate that Flan-MoE surpasses dense models in various experiment settings, including instruction-finetuning only and instruction-finetuning followed by task-specific finetuning. This highlights the crucial role of instruction-finetuning in enhancing the performance of MoE models. Notably, the largest model in the study, Flan-MoE-32B, outperforms Flan-PaLM-62B across four benchmarks while utilizing only one-third of the FLOPs. This success underscores the importance of rethinking the design of large-scale, high-performance language models within a framework of  The training data for all models includes 1,836 finetuning tasks derived from a combination of four mixtures: Muffin (comprising 80 tasks from previous work and 26 dialog/program synthesis tasks), T0-SF (193 tasks), NIV2 (1554 tasks), and CoT (9 reasoning tasks). Evaluations are conducted through zero-shot and few-shot assessments on held-out tasks not included in the finetuning data. Various benchmarks such as MMLU (featuring exam questions from diverse fields), BBH (challenging tasks from BIG-Bench), reasoning benchmarks like GSM8K and SVAMP/ASDIV focusing on math word problems and open-domain questions like StrategyQA are utilized for evaluation purposes. The study reports results for different benchmarks using direct prompting and chain-of-thought prompting approaches.