# What Matters for Model Merging at Scale?

**Authors:** Prateek Yadav, Tu Vu, Jonathan Lai, Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, Tsendsuren Munkhdalai  
**arXiv:** [2410.03617](https://arxiv.org/abs/2410.03617)  
**Date:** 2024-10-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.03617v1/

## Key points

- Model merging combines multiple expert models into a single, more capable model
- Benefits of model merging include reduced storage and serving costs, improved generalization, and support for decentralized model development
- Previous studies focused on small models, lacking understanding of how scaling model size impacts final merged model performance
- Recent works have demonstrated successful results for larger models up to 13B parameters but often neglect the effects of merging on generalization abilities
- This study systematically evaluates large-scale model merging by considering factors like base model quality and number of merged models on both held-in tasks and generalization performance
- Findings suggest that experts from strong base models lead to more effective merging processes; larger models facilitate easier merging; and merged models consistently improve generalization capabilities, often outperforming multitask trained ones when combining eight large expert models
- Larger models allow for better integration during the process while different methods exhibit similar behavior at larger scales
- This work provides valuable insights into the interplay between different factors affecting merged model performance, serving as a reference point for future research on large-scale model merging

## Summary

Model merging is a technique that combines multiple expert models into a single, more capable model. This offers benefits such as reduced storage and serving costs, improved generalization, and support for decentralized model development. Previous studies have explored model merging with small models but lack understanding of how scaling model size interacts with other key factors to impact the final merged model's performance. Recent works have demonstrated successful results for larger models up to 13B parameters but often focus on improving quality without considering the effects of merging on generalization abilities. In contrast, this study systematically evaluates the utility of large-scale model merging by examining various factors like base model quality and number of merged models on both held-in tasks and generalization performance. The experiments involve merging fully fine-tuned models using four popular methods across a range of sizes from 1B to 64B parameters and up to eight different expert models. The findings reveal insights about large-scale model merging - experts from strong base models lead to more effective merging processes; larger models facilitate easier merging; and merged models consistently improve generalization capabilities, often outperforming multitask trained ones when combining eight large expert models. Additionally, larger models allow for better integration during the process while different methods exhibit similar behavior at larger scales. This work serves as a reference point for future research on large-scale model merging by providing valuable insights into the interplay between different factors affecting merged model performance.