# Scaling Laws for Fine-Grained Mixture of Experts

**Authors:** Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michał Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóźdź, Piotr Sankowski, Marek Cygan, Sebastian Jaszczur  
**arXiv:** [2402.07871](https://arxiv.org/abs/2402.07871)  
**Date:** 2024-02-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.07871v1/

## Key points

- Authors explore Mixture of Experts (MoE) models to reduce computational cost of Large Language Models
- Introduce new hyperparameter called granularity for precise control over size of experts
- Establish scaling laws for fine-grained MoE models by incorporating training tokens, model size, and granularity adjustment
- Findings show MoE models outperform dense Transformers, efficiency gap widens with larger models and budgets
- Challenge common practice of setting expert sizes in MoE models to mirror feed-forward layers as suboptimal
- Study provides insights into optimizing MoE models for efficient large-scale language processing tasks

## Summary

In their work titled "Scaling Laws for Fine-Grained Mixture of Experts," authors Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michał Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóźdź, Piotr Sankowski, Marek Cygan and Sebastian Jaszczur delve into the analysis of Mixture of Experts (MoE) models as a solution to reduce the computational cost of Large Language Models. The study focuses on exploring the scaling properties of MoE models by introducing a new hyperparameter called granularity. This parameter allows for precise control over the size of the experts within the model. By incorporating an expanded range of variables such as training tokens and model size with granularity adjustment,the authors establish scaling laws for fine-grained MoE models. Through leveraging these laws,the authors are able to derive optimal training configurations based on a given computational budget. Their findings demonstrate that MoE models consistently outperform dense Transformers and reveal that the efficiency gap between dense and MoE models widens as model size and training budget scale up. Additionally,the authors challenge the common practice of setting expert sizes in MoE models to mirror feed-forward layers by showing that this approach is suboptimal across various computational budgets. Overall,this study provides valuable insights into optimizing MoE models for efficient large-scale language processing tasks.