# Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models

**Authors:** Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, Denny Zhou  
**arXiv:** [2305.14705](https://arxiv.org/abs/2305.14705)  
**Date:** 2023-05-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.14705v2/

## Key points

- Sheng Shen et al. propose a neural architecture design called Sparse Mixture-of-Experts (MoE) for Large Language Models (LLMs)
- Sparse MoE allows for the addition of learnable parameters without increasing inference cost
- Introduction of instruction tuning, a technique for training LLMs to follow instructions
- MoE models benefit more from instruction tuning compared to dense models
- Empirical studies conducted across three experimental setups:
- Direct fine-tuning on individual downstream tasks without instruction tuning shows MoE models underperform dense models with identical computational capacity
- Instruction tuning followed by in-context few-shot or zero-shot generalization significantly improves performance of MoE models compared to dense models
- Instruction tuning supplemented by further fine-tuning on individual downstream tasks leads to even better results
- FLAN-MOE-32B model surpasses FLAN-PALM-62B on four benchmark tasks while using only one third of the FLOPs
- Advancements showcased by FLAN-MOE inspire reevaluation of large scale language model design principles within task agnostic learning framework
- Combining Sparse Mixture-of Experts with instruction tuning enhances performance of Large Language Models while maintaining computational efficiency
- Task agnostic learning approaches like MoE models have significant potential for improving language understanding and generation capabilities in various applications.

## Summary

In their paper titled "Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models," Sheng Shen et al. propose a neural architecture design called Sparse Mixture-of-Experts (MoE) that allows for the addition of learnable parameters to Large Language Models (LLMs) without increasing inference cost. They also introduce instruction tuning, a technique for training LLMs to follow instructions. The authors advocate combining these two approaches and find that MoE models benefit more from instruction tuning compared to dense models. To evaluate the effectiveness of their proposed approach, the authors conduct empirical studies across three experimental setups. First, they perform direct fine-tuning on individual downstream tasks without instruction tuning. This scenario shows that MoE models underperform dense models with identical computational capacity. In the second scenario, instruction tuning is followed by in-context few-shot or zero-shot generalization on downstream tasks. This approach significantly improves the performance of MoE models compared to dense models. In the third scenario, instruction tuning is supplemented by further fine-tuning on individual downstream tasks, leading to even better results. The authors highlight their most powerful model, FLAN-MOE-32B, which surpasses the performance of FLAN-PALM-62B on four benchmark tasks while using only one third of the FLOPs (floating point operations). These advancements showcased by FLAN-MOE inspire a reevaluation of large scale language model design principles within the framework of task agnostic learning. Overall, this study demonstrates that combining Sparse Mixture-of Experts with instruction tuning can enhance the performance of Large Language Models while maintaining computational efficiency. The findings suggest that task agnostic learning approaches like MoE models have significant potential for improving language understanding and generation capabilities in various applications.