Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Another approach to the problem Not relevant

   Model soups need only one ingredient

   Alireza Abdollahpoorrostam, Nikolaos Dimitriadis, Adam Hazimeh et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet

   MonoSoup uses SVD on a single fine-tuned checkpoint to balance in-distribution accuracy and out-of-distribution robustness, avoiding the multiple checkpoints required by Model Soups.

   Abstract

   Fine-tuning large pre-trained models on a target distribution often improves in-distribution (ID) accuracy, but at the cost of out-of-distribution (OOD) robustness as representations specialize to the fine-tuning data. Weight-space ensembling methods, such as Model Soups, mitigate this effect by averaging multiple checkpoints, but they are computationally prohibitive, requiring the training and storage of dozens of fine-tuned models. In this paper, we introduce MonoSoup, a simple, data-free, hyperparameter-free, post-hoc method that achieves a strong ID-OOD balance using only a single checkpoint. Our method applies Singular Value Decomposition (SVD) to each layer's update and decomposes it i…

2. Close Broader context Not relevant

   Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfer

   Guo-Dong Du, Zitao Fang, Jing Li et al. · 2025 · Annual Meeting of the Association for Computational Linguistics · cited 22 times · Not summarized yet

   Neural Parameter Search prunes fine-tuned models and combines them with pretrained weights to improve transfer and reduce redundancy, rather than averaging fine-tuned models.

   Abstract

   Foundation models and their checkpoints have significantly advanced deep learning, boosting performance across various applications. However, fine-tuned models often struggle outside their specific domains and exhibit considerable redundancy. Recent studies suggest that combining a pruned fine-tuned model with the original pre-trained model can mitigate forgetting, reduce interference when merging model parameters across tasks, and improve compression efficiency. In this context, developing an effective pruning strategy for fine-tuned models is crucial. Leveraging the advantages of the task vector mechanism, we preprocess fine-tuned models by calculating the differences between them and the …

3. Close Builds on this paper Not relevant

   Quantized Model Soup Shake-Up: Weight Perturbation for Enhanced Ensemble Diversity

   Jinwook Chung, Sungyeop Jung, Weronika Czorapinska et al. · 2026 · Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · cited 1 time · Not summarized yet

   Quantized Model Soup Shake-Up studies how quantization affects model soups and perturbs weights to restore diversity among quantized candidates.

   Abstract

   Model soup, averaging the weights of multiple fine-tuned models, delivers ensemble-level accuracy at single-model inference cost, but its success requires both linear mode connectivity (LMC) and sufficient diversity among candidates. We study these two requirements under quantization-aware training (QAT). First, we show analytically and empirically that fine-tuning from a sufficiently converged QAT checkpoint, which we call the QAT anchor state, preserves linear mode connectivity by keeping models within the same loss basin. Second, we identify Ensemble Degeneracy, where the many-to-one mapping of the quantizer collapses independently fine-tuned models into nearly identical quantized represe…

4. Close Builds on this paper Not relevant

   Self-Soupervision: Cooking Model Soups without Labels

   A. Fuller, James R. Green, Evan Shelhamer · 2026 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet

   Self-Soupervision extends model soups to self-supervised learning, using unlabeled data to improve transfer and robustness.

   Abstract

   Model soups are strange and strangely effective combinations of parameters. They take a model (the stock), fine-tune it into multiple models (the ingredients), and then mix their parameters back into one model (the soup) to improve predictions. While all known soups require supervised learning, and optimize the same loss on labeled data, our recipes for Self-Soupervision generalize soups to self-supervised learning (SSL). Our Self-Souping lets us flavor ingredients on new data sources, e.g. from unlabeled data from a task for transfer or from a shift for robustness. We show that Self-Souping on corrupted test data, then fine-tuning back on uncorrupted train data, boosts robustness by +3.5% (…

5. Close Another approach to the problem Not relevant

   Model Stock: All we need is just a few fine-tuned models

   Dong-Hwan Jang, Sangdoo Yun, Dongyoon Han · 2024 · European Conference on Computer Vision · cited 115 times · Not summarized yet

   Model Stock approximates a center-close weight using two fine-tuned models, reducing the number needed to obtain strong in- and out-of-distribution performance.

   Abstract

   This paper introduces an efficient fine-tuning method for large pre-trained models, offering strong in-distribution (ID) and out-of-distribution (OOD) performance. Breaking away from traditional practices that need a multitude of fine-tuned models for averaging, our approach employs significantly fewer models to achieve final weights yet yield superior accuracy. Drawing from key insights in the weight space of fine-tuned weights, we uncover a strong link between the performance and proximity to the center of weight space. Based on this, we introduce a method that approximates a center-close weight using only two fine-tuned models, applicable during or after training. Our innovative layer-wis…

6. Close Groundwork for this paper Cited by this paper Used in depth by this paper A method it uses Not relevant

   Averaging Weights Leads to Wider Optima and Better Generalization

   Pavel Izmailov, Dmitrii Podoprikhin, T. Garipov et al. · 2018 · Conference on Uncertainty in Artificial Intelligence · cited 2,254 times · Not summarized yet

   Stochastic Weight Averaging averages points along one SGD trajectory and connects weight averaging with flatter solutions and better generalization.

   Abstract

   Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence. We show that simple averaging of multiple points along the trajectory of SGD, with a cyclical or constant learning rate, leads to better generalization than conventional training. We also show that this Stochastic Weight Averaging (SWA) procedure finds much flatter solutions than SGD, and approximates the recent Fast Geometric Ensembling (FGE) approach with a single model. Using SWA we achieve notable improvement in test accuracy over conventional SGD training on a range of state-of-the-art residual networks, PyramidNets, DenseNets, an…

7. Close Groundwork for this paper Cited by this paper Not relevant

   Distilling the Knowledge in a Neural Network

   Geoffrey Hinton, Oriol Vinyals, Jeff Dean · 2015 · Machine Learning · preprint, not peer reviewed · cited 27,469 times · AI summary available

   Distilling the Knowledge in a Neural Network compresses ensemble predictions into a single model to reduce deployment costs.

   Abstract

   A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we …

8. Close* Builds on this paper Cites this paper Not relevant

   Branch Soup: Improving vision foundation models via averaging adapter weights of branched checkpoints

   Youmin Park, Yeonguk Yu, Yunjae Heo et al. · 2026 · Neurocomputing · Not summarized yet

   Branch Soup averages adapter weights from branched checkpoints to improve vision foundation models.

Prior work it builds on

1. Close Broader context Used in depth by this paper A method it uses Not relevant

   No One Representation to Rule Them All: Overlapping Features of Training Methods

   Raphael Gontijo Lopes, Y. Dauphin, E. Cubuk · 2021 · International Conference on Learning Representations · cited 75 times · Not summarized yet

   No One Representation to Rule Them All empirically studies overlapping features and prediction differences across models trained with different methods.

   Abstract

   Despite being able to capture a range of features of the data, high accuracy models trained with supervision tend to make similar predictions. This seemingly implies that high-performing models share similar biases regardless of training methodology, which would limit ensembling benefits and render low-accuracy models as having little practical use. Against this backdrop, recent work has developed quite different training techniques, such as large-scale contrastive learning, yielding competitively high accuracy on generalization and robustness benchmarks. This motivates us to revisit the assumption that models necessarily learn similar functions. We conduct a large-scale empirical study of m…

2. Related Groundwork for this paper Used in depth by this paper A method it uses Not relevant

   Robust fine-tuning of zero-shot models

   Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim et al. · 2021 · Computer Vision and Pattern Recognition · cited 1,101 times · Not summarized yet

   Robust fine-tuning of zero-shot models introduces WiSE-FT, averaging zero-shot and fine-tuned weights to improve accuracy under distribution shift.

   Abstract

   Large pre-trained models such as CLIP or ALIGN offer consistent accuracy across a range of data distributions when performing zero-shot inference (i.e., without fine-tuning on a specific dataset). Although existing fine-tuning methods substantially improve accuracy on a given target distribution, they often reduce robustness to distribution shifts. We address this tension by introducing a simple and effective method for improving robustness while fine-tuning: ensembling the weights of the zero-shot and fine-tuned models (WiSE-FT). Compared to standard fine-tuning, WiSE-FT provides large accuracy improvements under distribution shift, while preserving high accuracy on the target distribution.…

3. Related Broader context Used in depth by this paper A method it uses Not relevant

   Combined Scaling for Zero-shot Transfer Learning

   Hieu Pham, Zi-Hang Dai, Golnaz Ghiasi et al. · 2021 · Neurocomputing · cited 253 times · Not summarized yet

   Combined Scaling for Zero-shot Transfer Learning scales contrastive pretraining to improve ImageNet accuracy and robustness without labeled ImageNet examples.

   Abstract

   We present a combined scaling method - named BASIC - that achieves 85.7% top-1 accuracy on the ImageNet ILSVRC-2012 validation set without learning from any labeled ImageNet example. This accuracy surpasses best published similar models - CLIP and ALIGN - by 9.3%. Our BASIC model also shows significant improvements in robustness benchmarks. For instance, on 5 test sets with natural distribution shifts such as ImageNet-{A,R,V2,Sketch} and ObjectNet, our model achieves 84.3% top-1 average accuracy, only a small drop from its original ImageNet accuracy. To achieve these results, we scale up the contrastive learning framework of CLIP and ALIGN in three dimensions: data size, model size, and batc…

4. Related Broader context A method it uses Not relevant

   Zoo-Tuning: Adaptive Transfer from a Zoo of Models

   Yang Shu, Zhi Kou, Zhangjie Cao et al. · 2021 · International Conference on Machine Learning · cited 50 times · Not summarized yet

   Zoo-Tuning adaptively aggregates aligned parameters from pretrained models in a model zoo for transfer to target tasks.

   Abstract

   With the development of deep networks on various large-scale datasets, a large zoo of pretrained models are available. When transferring from a model zoo, applying classic single-model based transfer learning methods to each source model suffers from high computational burden and cannot fully utilize the rich knowledge in the zoo. We propose \emph{Zoo-Tuning} to address these challenges, which learns to adaptively transfer the parameters of pretrained models to the target task. With the learnable channel alignment layer and adaptive aggregation layer, Zoo-Tuning \emph{adaptively aggregates channel aligned pretrained parameters} to derive the target model, which promotes knowledge transfer by…

5. Related Groundwork for this paper Used in depth by this paper A method it uses Not relevant

   Big Transfer (BiT): General Visual Representation Learning

   Alexander Kolesnikov, Lucas Beyer, Xiao-Hua Zhai et al. · 2019 · European Conference on Computer Vision · cited 1,427 times · Not summarized yet

   Big Transfer establishes a pretraining-and-fine-tuning recipe for visual representations and reports strong results across more than 20 datasets.

   Abstract

   Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. BiT performs well across a surprisingly wide range of data regimes -- from 1 example per class to 1M total examples. BiT achieves 87.5% top-1 accuracy on ILSVRC-2012, 99.4% on CIFAR-10, and 76.3% on the 19…

Papers that cite it

1. Close Builds on this paper Uses this paper in depth Not relevant

   Trajectory Soup: Pushing the Compute-Scaling Frontier of LLM Mid-training via Diverse Trajectories

   Zhehao Huang, Chang-Xin Tian, Qing-Yuan Yang et al. · 2026 · Not summarized yet

   Trajectory Soup averages diverse optimization trajectories during LLM mid-training to improve capabilities within a single model.

   Abstract

   Mid-training equips pretrained large language models with specialized and reasoning capabilities, but the returns of this stage are bounded since additional serial compute yields little further downstream improvement and can even degrade some capabilities, which places a practical ceiling on how much compute mid-training absorbs. We revisit how this compute should be allocated to a single run or multiple similar optimizations. We find that branches forked from a shared checkpoint under various controlled recipe reaches measurably different regions of parameter space, and establish a form of compatible diversity that extending one run cannot supply. Therefore, we introduce Trajectory Soup, wh…

2. Close Broader context Not relevant

   Finetune like you pretrain: Improved finetuning of zero-shot vision models

   Sachin Goyal, Ananya Kumar, Sankalp Garg et al. · 2023 · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conferen · cited 99 times · Not summarized yet

   Finetune like you pretrain improves transfer of zero-shot vision models by continuing contrastive training with image embeddings and class-descriptive prompts.

   Abstract

   Finetuning image-text models such as CLIP achieves state-of-the-art accuracies on a variety of benchmarks. However, recent works (Kumar et al., 2022; Wortsman et al., 2021) have shown that even subtle differences in the finetuning process can lead to surprisingly large differences in the final performance, both for in-distribution (ID) and out-of-distribution (OOD) data. In this work, we show that a natural and simple approach of mimicking contrastive pretraining consistently outperforms alternative finetuning approaches. Specifically, we cast downstream class labels as text prompts and continue optimizing the contrastive loss between image embeddings and class-descriptive prompt embeddings …

3. Related Broader context Uses this paper in depth Not relevant

   Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs

   Nikita Kozodoi, Zainab Afolabi, Jack Butler · 2026 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet

   Are we Merging the Right Models? examines how expert training duration affects multi-task model merging and finds outcomes depend on the merging method.

   Abstract

   Multi-task model merging combines separately trained expert models into a single model that handles all tasks without co-training. Standard practice merges experts at their optimal validation loss. We challenge this convention by systematically studying how training duration of domain experts affects the quality of the merged model. We fine-tune experts on five domains (Math, Code, Instruction Following, Multilingual, and Safety) across three model sizes (Qwen 3.5 0.8B, 2B, and 4B), saving checkpoints from 25% to 500% of the optimal training steps and evaluating five merging methods at each duration. Our findings reveal a striking method-dependent pattern: simple averaging degrades sharply w…

4. Related Broader context Uses this paper in depth Not relevant

   Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging

   Minsik Choi, Geewook Kim · 2026 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet

   Decentralized Instruction Tuning studies conflict-aware splitting and parameter merging for independently trained instruction-tuning models.

   Abstract

   Instruction tuning aligns large language models, including multimodal ones, with diverse user intents, but scaling to heterogeneous mixtures is hindered by gradient interference and bandwidth-heavy synchronization. We ask whether these two bottlenecks can be addressed jointly by training parts of the mixture independently and reconciling them once in parameter space. We develop a local quadratic theory inside a shared flat basin that yields three results: weight merging produces a curvature-weighted variance reduction; PCA-aligned conflict splitting maximizes this gain along high-curvature directions; and merging additionally acts as spectral filtering with implicit norm regularization. Thes…

5. Close Broader context Not relevant

   Lost in the Non-convex Loss Landscape: How to Fine-tune the Large Time Series Model?

   Xu Zhang, Peang Wang, Wei Wang · 2026 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet

   Lost in the Non-convex Loss Landscape proposes smoothed full fine-tuning for large time series models with poorly conditioned loss landscapes.

   Abstract

   Recently, large time series models (LTSMs) have gained increasing attention due to their similarities to large language models, including flexible context length, scalability, and task generality, outperforming advanced task-specific models. However, prior studies indicate that pre-trained LTSMs may exhibit a poorly conditioned non-convex loss landscape, leading to limited trainability. As a result, direct fine-tuning tends to cause overfitting and suboptimal performance, sometimes even worse than training from scratch, substantially diminishing the benefits of pre-training. To overcome this limitation, we propose Smoothed Full Fine-tuning (SFF), a novel fine-tuning technology. Specifically,…