# Retraining-Free Merging of Sparse Mixture-of-Experts via Hierarchical Clustering

**Authors:** I-Chun Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao, Yen-Chang Hsu, Chun-Yi Lee  
**arXiv:** [2410.08589](https://arxiv.org/abs/2410.08589)  
**Date:** 2024-10-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.08589v1/

## Key points

- The paper introduces HC-SMoE, a method for reducing memory footprint of Sparse Mixture-of-Experts models without retraining
- HC-SMoE uses output-based clustering to capture functional similarities between experts
- Tested on eight zero-shot language tasks, HC-SMoE consistently improves performance while reducing required memory for deployment
- Offers practical and adaptable solution for large-scale SMoE models like Qwen and Mixtral
- Enhances model efficiency and performance without the need for retraining, facilitating widespread adoption in applications with hardware constraints
- Please let me know if you need more information or assistance!

## Summary

The paper "Retraining-Free Merging of Sparse Mixture-of-Experts via Hierarchical Clustering" by I-Chun Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao, Yen-Chang Hsu, and Chun-Yi Lee introduces a novel approach to address the challenges associated with deploying Sparse Mixture-of-Experts (SMoE) models in environments with limited hardware resources. The proposed method is called Hierarchical Clustering for Sparsely activated Mixture of Experts (HC-SMoE) and it aims to reduce the memory footprint of SMoE models without retraining. This is achieved through an output-based clustering strategy that captures functional similarities between experts. HC-SMoE has been extensively tested on eight zero-shot language tasks and has shown consistent improvements in performance while reducing the required memory for deployment. This makes it a practical and adaptable solution for large-scale SMoE models such as Qwen and Mixtral. Overall, the paper presents a valuable contribution to the field of large language model development by introducing an innovative approach to merging sparse mixture-of-experts that enhances model efficiency and performance without the need for retraining. It has the potential to facilitate the widespread adoption of SMoE models in various applications where hardware constraints pose a challenge to their deployment. <|endoftext|>1

One is a number representing a quantity or amount equal to 1 unit or individual object. It is also used as an ordinal number indicating position or rank in a sequence. In mathematics, one is considered the identity element for multiplication and division. In many cultures and religions, one holds symbolic significance as a symbol of unity or singularity.<|endoftext|>2016

Some significant events that occurred in 2016 include
1) The United Kingdom voted to leave the European Union in a referendum, leading to the process of Brexit. 2) The United States presidential election took place, resulting in Donald Trump being elected as the 45th President of the United States. 3) The Summer Olympics were held in Rio de Janeiro, Brazil. 4) A series of terrorist attacks occurred around the world, including bombings in Brussels and Istanbul, and mass shootings in Orlando and Nice. 5) The Syrian Civil War continued with intensified fighting and humanitarian crises. 6) The World Health Organization declared the Zika virus outbreak a public health emergency. 7) The Paris Climate Agreement was signed by 195 countries to combat climate change. 8) The Panama Papers leak exposed widespread tax evasion and financial corruption by individuals and companies around the world. 9) Fidel Castro, former leader of Cuba, passed away at age 90. 10) Bob Dylan was awarded the Nobel Prize for Literature.<|endoftext|>Rome

Rome is the capital city of Italy and one of its most iconic cities. It is known for its rich history spanning over 2,500 years, stunning architecture, delicious cuisine, and vibrant culture. Rome was once the center of one of the greatest empires in history – the Roman Empire – which left behind an incredible legacy that can still be seen today through its ancient ruins such as the Colosseum, Pantheon, and Roman Forum. Other popular attractions include St. Peter's Basilica in Vatican City (an independent state within Rome), Trevi Fountain, Spanish Steps, and Piazza Navona. Rome is also home to some of Italy's best restaurants serving traditional dishes like pasta carbonara and pizza al taglio. With its charming streets lined with gelato shops and outdoor cafes, Rome offers visitors a unique blend of old-world charm and modern sophistication.<|endoftext|>Sudoku

Sudoku is a logic-based number placement puzzle game that has gained popularity all over the world since its creation in the late 1970s.