Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

1. Très proche Pas pertinent

   Model soups need only one ingredient

   Alireza Abdollahpoorrostam, Nikolaos Dimitriadis, Adam Hazimeh et al. · 2026 · arXiv.org · preprint, non relu par les pairs · cité 1 fois · Pas encore résumé

   Résumé scientifique

   Fine-tuning large pre-trained models on a target distribution often improves in-distribution (ID) accuracy, but at the cost of out-of-distribution (OOD) robustness as representations specialize to the fine-tuning data. Weight-space ensembling methods, such as Model Soups, mitigate this effect by averaging multiple checkpoints, but they are computationally prohibitive, requiring the training and storage of dozens of fine-tuned models. In this paper, we introduce MonoSoup, a simple, data-free, hyperparameter-free, post-hoc method that achieves a strong ID-OOD balance using only a single checkpoint. Our method applies Singular Value Decomposition (SVD) to each layer's update and decomposes it i…

2. Proche Pas pertinent

   Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfer

   Guo-Dong Du, Zitao Fang, Jing Li et al. · 2025 · Annual Meeting of the Association for Computational Linguistics · cité 22 fois · Pas encore résumé

   Résumé scientifique

   Foundation models and their checkpoints have significantly advanced deep learning, boosting performance across various applications. However, fine-tuned models often struggle outside their specific domains and exhibit considerable redundancy. Recent studies suggest that combining a pruned fine-tuned model with the original pre-trained model can mitigate forgetting, reduce interference when merging model parameters across tasks, and improve compression efficiency. In this context, developing an effective pruning strategy for fine-tuned models is crucial. Leveraging the advantages of the task vector mechanism, we preprocess fine-tuned models by calculating the differences between them and the …

3. Proche Pas pertinent

   Quantized Model Soup Shake-Up: Weight Perturbation for Enhanced Ensemble Diversity

   Jinwook Chung, Sungyeop Jung, Weronika Czorapinska et al. · 2026 · Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · cité 1 fois · Pas encore résumé

   Résumé scientifique

   Model soup, averaging the weights of multiple fine-tuned models, delivers ensemble-level accuracy at single-model inference cost, but its success requires both linear mode connectivity (LMC) and sufficient diversity among candidates. We study these two requirements under quantization-aware training (QAT). First, we show analytically and empirically that fine-tuning from a sufficiently converged QAT checkpoint, which we call the QAT anchor state, preserves linear mode connectivity by keeping models within the same loss basin. Second, we identify Ensemble Degeneracy, where the many-to-one mapping of the quantizer collapses independently fine-tuned models into nearly identical quantized represe…

4. Proche Pas pertinent

   Self-Soupervision: Cooking Model Soups without Labels

   A. Fuller, James R. Green, Evan Shelhamer · 2026 · arXiv.org · preprint, non relu par les pairs · cité 1 fois · Pas encore résumé

   Résumé scientifique

   Model soups are strange and strangely effective combinations of parameters. They take a model (the stock), fine-tune it into multiple models (the ingredients), and then mix their parameters back into one model (the soup) to improve predictions. While all known soups require supervised learning, and optimize the same loss on labeled data, our recipes for Self-Soupervision generalize soups to self-supervised learning (SSL). Our Self-Souping lets us flavor ingredients on new data sources, e.g. from unlabeled data from a task for transfer or from a shift for robustness. We show that Self-Souping on corrupted test data, then fine-tuning back on uncorrupted train data, boosts robustness by +3.5% (…

5. Proche Pas pertinent

   Model Stock: All we need is just a few fine-tuned models

   Dong-Hwan Jang, Sangdoo Yun, Dongyoon Han · 2024 · European Conference on Computer Vision · cité 115 fois · Pas encore résumé

   Résumé scientifique

   This paper introduces an efficient fine-tuning method for large pre-trained models, offering strong in-distribution (ID) and out-of-distribution (OOD) performance. Breaking away from traditional practices that need a multitude of fine-tuned models for averaging, our approach employs significantly fewer models to achieve final weights yet yield superior accuracy. Drawing from key insights in the weight space of fine-tuned weights, we uncover a strong link between the performance and proximity to the center of weight space. Based on this, we introduce a method that approximates a center-close weight using only two fine-tuned models, applicable during or after training. Our innovative layer-wis…

6. Proche Cité par cet article Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   Averaging Weights Leads to Wider Optima and Better Generalization

   Pavel Izmailov, Dmitrii Podoprikhin, T. Garipov et al. · 2018 · Conference on Uncertainty in Artificial Intelligence · cité 2 254 fois · Pas encore résumé

   Résumé scientifique

   Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence. We show that simple averaging of multiple points along the trajectory of SGD, with a cyclical or constant learning rate, leads to better generalization than conventional training. We also show that this Stochastic Weight Averaging (SWA) procedure finds much flatter solutions than SGD, and approximates the recent Fast Geometric Ensembling (FGE) approach with a single model. Using SWA we achieve notable improvement in test accuracy over conventional SGD training on a range of state-of-the-art residual networks, PyramidNets, DenseNets, an…

7. Proche Cité par cet article Pas pertinent

   Distilling the Knowledge in a Neural Network

   Geoffrey Hinton, Oriol Vinyals, Jeff Dean · 2015 · Machine Learning · preprint, non relu par les pairs · cité 27 469 fois · Résumé IA disponible

   Résumé scientifique

   A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we …

8. Proche* Cite cet article Pas pertinent

   Branch Soup: Improving vision foundation models via averaging adapter weights of branched checkpoints

   Youmin Park, Yeonguk Yu, Yunjae Heo et al. · 2026 · Neurocomputing · Pas encore résumé

Les travaux antérieurs sur lesquels il s’appuie

1. Proche Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   No One Representation to Rule Them All: Overlapping Features of Training Methods

   Raphael Gontijo Lopes, Y. Dauphin, E. Cubuk · 2021 · International Conference on Learning Representations · cité 75 fois · Pas encore résumé

   Résumé scientifique

   Despite being able to capture a range of features of the data, high accuracy models trained with supervision tend to make similar predictions. This seemingly implies that high-performing models share similar biases regardless of training methodology, which would limit ensembling benefits and render low-accuracy models as having little practical use. Against this backdrop, recent work has developed quite different training techniques, such as large-scale contrastive learning, yielding competitively high accuracy on generalization and robustness benchmarks. This motivates us to revisit the assumption that models necessarily learn similar functions. We conduct a large-scale empirical study of m…

2. Apparenté Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   Robust fine-tuning of zero-shot models

   Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim et al. · 2021 · Computer Vision and Pattern Recognition · cité 1 101 fois · Pas encore résumé

   Résumé scientifique

   Large pre-trained models such as CLIP or ALIGN offer consistent accuracy across a range of data distributions when performing zero-shot inference (i.e., without fine-tuning on a specific dataset). Although existing fine-tuning methods substantially improve accuracy on a given target distribution, they often reduce robustness to distribution shifts. We address this tension by introducing a simple and effective method for improving robustness while fine-tuning: ensembling the weights of the zero-shot and fine-tuned models (WiSE-FT). Compared to standard fine-tuning, WiSE-FT provides large accuracy improvements under distribution shift, while preserving high accuracy on the target distribution.…

3. Apparenté Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   Combined Scaling for Zero-shot Transfer Learning

   Hieu Pham, Zi-Hang Dai, Golnaz Ghiasi et al. · 2021 · Neurocomputing · cité 253 fois · Pas encore résumé

   Résumé scientifique

   We present a combined scaling method - named BASIC - that achieves 85.7% top-1 accuracy on the ImageNet ILSVRC-2012 validation set without learning from any labeled ImageNet example. This accuracy surpasses best published similar models - CLIP and ALIGN - by 9.3%. Our BASIC model also shows significant improvements in robustness benchmarks. For instance, on 5 test sets with natural distribution shifts such as ImageNet-{A,R,V2,Sketch} and ObjectNet, our model achieves 84.3% top-1 average accuracy, only a small drop from its original ImageNet accuracy. To achieve these results, we scale up the contrastive learning framework of CLIP and ALIGN in three dimensions: data size, model size, and batc…

4. Apparenté Une méthode qu’il utilise Pas pertinent

   Zoo-Tuning: Adaptive Transfer from a Zoo of Models

   Yang Shu, Zhi Kou, Zhangjie Cao et al. · 2021 · International Conference on Machine Learning · cité 50 fois · Pas encore résumé

   Résumé scientifique

   With the development of deep networks on various large-scale datasets, a large zoo of pretrained models are available. When transferring from a model zoo, applying classic single-model based transfer learning methods to each source model suffers from high computational burden and cannot fully utilize the rich knowledge in the zoo. We propose \emph{Zoo-Tuning} to address these challenges, which learns to adaptively transfer the parameters of pretrained models to the target task. With the learnable channel alignment layer and adaptive aggregation layer, Zoo-Tuning \emph{adaptively aggregates channel aligned pretrained parameters} to derive the target model, which promotes knowledge transfer by…

5. Apparenté Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   Big Transfer (BiT): General Visual Representation Learning

   Alexander Kolesnikov, Lucas Beyer, Xiao-Hua Zhai et al. · 2019 · European Conference on Computer Vision · cité 1 427 fois · Pas encore résumé

   Résumé scientifique

   Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. BiT performs well across a surprisingly wide range of data regimes -- from 1 example per class to 1M total examples. BiT achieves 87.5% top-1 accuracy on ILSVRC-2012, 99.4% on CIFAR-10, and 76.3% on the 19…

Les articles qui le citent

1. Proche Utilise cet article en profondeur Pas pertinent

   Trajectory Soup: Pushing the Compute-Scaling Frontier of LLM Mid-training via Diverse Trajectories

   Zhehao Huang, Chang-Xin Tian, Qing-Yuan Yang et al. · 2026 · Pas encore résumé

   Résumé scientifique

   Mid-training equips pretrained large language models with specialized and reasoning capabilities, but the returns of this stage are bounded since additional serial compute yields little further downstream improvement and can even degrade some capabilities, which places a practical ceiling on how much compute mid-training absorbs. We revisit how this compute should be allocated to a single run or multiple similar optimizations. We find that branches forked from a shared checkpoint under various controlled recipe reaches measurably different regions of parameter space, and establish a form of compatible diversity that extending one run cannot supply. Therefore, we introduce Trajectory Soup, wh…

2. Proche Pas pertinent

   Finetune like you pretrain: Improved finetuning of zero-shot vision models

   Sachin Goyal, Ananya Kumar, Sankalp Garg et al. · 2023 · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conferen · cité 99 fois · Pas encore résumé

   Résumé scientifique

   Finetuning image-text models such as CLIP achieves state-of-the-art accuracies on a variety of benchmarks. However, recent works (Kumar et al., 2022; Wortsman et al., 2021) have shown that even subtle differences in the finetuning process can lead to surprisingly large differences in the final performance, both for in-distribution (ID) and out-of-distribution (OOD) data. In this work, we show that a natural and simple approach of mimicking contrastive pretraining consistently outperforms alternative finetuning approaches. Specifically, we cast downstream class labels as text prompts and continue optimizing the contrastive loss between image embeddings and class-descriptive prompt embeddings …

3. Apparenté Utilise cet article en profondeur Pas pertinent

   Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs

   Nikita Kozodoi, Zainab Afolabi, Jack Butler · 2026 · arXiv.org · preprint, non relu par les pairs · cité 1 fois · Pas encore résumé

   Résumé scientifique

   Multi-task model merging combines separately trained expert models into a single model that handles all tasks without co-training. Standard practice merges experts at their optimal validation loss. We challenge this convention by systematically studying how training duration of domain experts affects the quality of the merged model. We fine-tune experts on five domains (Math, Code, Instruction Following, Multilingual, and Safety) across three model sizes (Qwen 3.5 0.8B, 2B, and 4B), saving checkpoints from 25% to 500% of the optimal training steps and evaluating five merging methods at each duration. Our findings reveal a striking method-dependent pattern: simple averaging degrades sharply w…

4. Apparenté Utilise cet article en profondeur Pas pertinent

   Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging

   Minsik Choi, Geewook Kim · 2026 · arXiv.org · preprint, non relu par les pairs · cité 2 fois · Pas encore résumé

   Résumé scientifique

   Instruction tuning aligns large language models, including multimodal ones, with diverse user intents, but scaling to heterogeneous mixtures is hindered by gradient interference and bandwidth-heavy synchronization. We ask whether these two bottlenecks can be addressed jointly by training parts of the mixture independently and reconciling them once in parameter space. We develop a local quadratic theory inside a shared flat basin that yields three results: weight merging produces a curvature-weighted variance reduction; PCA-aligned conflict splitting maximizes this gain along high-curvature directions; and merging additionally acts as spectral filtering with implicit norm regularization. Thes…

5. Proche Pas pertinent

   Lost in the Non-convex Loss Landscape: How to Fine-tune the Large Time Series Model?

   Xu Zhang, Peang Wang, Wei Wang · 2026 · arXiv.org · preprint, non relu par les pairs · cité 2 fois · Pas encore résumé

   Résumé scientifique

   Recently, large time series models (LTSMs) have gained increasing attention due to their similarities to large language models, including flexible context length, scalability, and task generality, outperforming advanced task-specific models. However, prior studies indicate that pre-trained LTSMs may exhibit a poorly conditioned non-convex loss landscape, leading to limited trainability. As a result, direct fine-tuning tends to cause overfitting and suboptimal performance, sometimes even worse than training from scratch, substantially diminishing the benefits of pre-training. To overcome this limitation, we propose Smoothed Full Fine-tuning (SFF), a novel fine-tuning technology. Specifically,…