# EdgeMoE: Fast On-Device Inference of MoE-based Large Language Models

**Authors:** Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, Mengwei Xu  
**arXiv:** [2308.14352](https://arxiv.org/abs/2308.14352)  
**Date:** 2023-08-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.14352v1/

## Key points

- EdgeMoE is an on-device inference engine for mixture-of-expert (MoE) Large Language Models (LLMs)
- LLMs have exceptional capabilities in machine learning tasks
- Transitioning LLMs from data centers to edge devices can enhance privacy and availability
- Large parameter sizes of LLMs result in impractical runtime costs on edge devices
- EdgeMoE strategically partitions the model across the storage hierarchy for memory and computational efficiency
- Non-expert weights are stored in device memory, while expert weights are kept in external storage and fetched into memory only when activated
- Expert weights are infrequently accessed due to sparse activation patterns
- EdgeMoE uses expert-wise bitwidth adaptation to reduce expert weight size without significant accuracy loss
- Expert management predicts which experts will be activated in advance and preloads them into the compute-I/O pipeline
- Empirical evaluations show that EdgeMoE achieves substantial memory savings and performance improvements compared to baseline solutions
- EdgeMoE provides a specialized on-device inference engine tailored for MoE-based models
- Strategic partitioning and innovative techniques improve efficiency, performance, and minimize runtime costs.

## Summary

EdgeMoE is an on-device inference engine designed for mixture-of-expert (MoE) Large Language Models (LLMs), which are known for their exceptional capabilities in various machine learning tasks. The transition of LLMs from data centers to edge devices can enhance privacy and availability but is hindered by the large parameter sizes of these models, resulting in impractical runtime costs. To address these issues, EdgeMoE strategically partitions the model across the storage hierarchy to achieve memory and computational efficiency. Non-expert weights are stored in the device's memory while expert weights are kept in external storage and fetched into memory only when they are activated. This design is based on the insight that expert weights, although voluminous, are infrequently accessed due to sparse activation patterns. Additionally, EdgeMoE incorporates two innovative techniques to mitigate the overhead associated with expert input/output swapping: expert-wise bitwidth adaptation to reduce the size of expert weights without significant accuracy loss and expert management to predict which experts will be activated in advance and preloads them into the compute-I/O pipeline. Empirical evaluations conducted on well-established MoE LLMs and various edge devices demonstrate that EdgeMoE achieves substantial memory savings and performance improvements compared to competitive baseline solutions. Overall, EdgeMoE addresses the challenges of deploying LLMs on edge devices by providing a specialized on-device inference engine tailored for MoE-based models. Its strategic partitioning of model components and innovative techniques result in improved efficiency and performance while minimizing runtime costs.