# Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts

**Authors:** Soumajyoti Sarkar, Yuxin Tang, Sheng Zha  
**arXiv:** [2608.10605](https://arxiv.org/abs/2608.10605)  
**Date:** 2026-08-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2608.10605v1/

## Key points

- The study introduces MOSAIC, a framework integrating model architecture and systems co-design for optimization challenges.
- MOSAIC estimates Model FLOPs Utilization (MFU), communication costs, memory footprint, and optimal parallel layout.
- Sparse Mixture-of-Experts (MoE) language models are specifically focused on, analyzing factors like expert count and routing sparsity.
- Scaling law analysis on sparse MoE models shows the impact of varying sparsity levels on model performance.
- Efficiency in large-scale language model training is improved by optimizing both model architecture and hardware systems together.
- An optimal sparsity level for MoE models can be achieved under hardware limitations consideration.
- Related research efforts align model dimensions with computational boundaries for efficiency and accuracy improvement.
- Hybrid architectures designed for compute optimality and improved recall rates are mentioned in the study.
- Joint optimization of model architecture and hardware systems is advocated for superior performance outcomes in language modeling tasks.

## Summary

The study "Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts" delves into the traditional approach of disjointed decision-making in large-scale pretraining processes. The research introduces MOSAIC, a framework that integrates model architecture and systems co-design as an optimization challenge. By combining a predictive scaling law with a calibrated performance model, MOSAIC estimates Model FLOPs Utilization (MFU), communication costs, memory footprint, and optimal parallel layout. Specifically focusing on sparse Mixture-of-Experts (MoE) language models, the study explores how factors like expert count, routing sparsity, and other MoE layer dimensions impact both loss function optimization and system efficiency. Through fitting a scaling law on sparse MoE models trained on text data, the researchers analyze the effects of varying sparsity levels on model performance. is a crucial step in large-scale language model training processes. However,(MoE) models have been shown to outperform traditional approaches by incorporating specialized experts for different tasks. This leads to , where both model architecture and hardware systems are optimized together for maximum efficiency. MOSAIC provides a solution by estimating key metrics such as MFU and communication costs to determine an optimal parallel layout for efficient training. The study also highlights the impact of various  dimensions on both loss function optimization and system efficiency. Through experiments on sparse MoE models trained on text data with active parameters ranging from 104 million to 2.7 billion and total model sizes up to 79 billion parameters, the researchers found that an efficiency-agnostic model-FLOPs budget does not yield an interior optimal sparsity level. Instead, the compute optimum is situated at the upper boundary of the data support range. Interestingly, MOSAIC reveals that an optimal sparsity level for MoE models can be achieved under , highlighting the importance of considering hardware limitations in model design. This further emphasizes the need for  in both architecture and systems co-design for cutting-edge language model training. The study also references related research efforts that align model dimensions with computational boundaries for improved efficiency and accuracy. It also mentions hybrid architectures designed for compute optimality and improved recall rates, as well as efforts focused on co-designing transformers and accelerators to meet environmental objectives. In conclusion, "Compute-Optimal Is Not Cluster-Optimal" advocates for a holistic approach to decision-making in pretraining processes by emphasizing joint optimization of model architecture and hardware systems. By doing so, superior performance outcomes can be achieved in frontier language modeling tasks.