# Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch

**Authors:** Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, Yongbin Li  
**arXiv:** [2311.03099](https://arxiv.org/abs/2311.03099)  
**Date:** 2023-11-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.03099v1/

## Key points

- Authors explore how Language Models (LMs) can enhance capabilities through parameter assimilation from similar models without retraining or additional GPUs
- Study focuses on encoder- or decoder-based LMs and effectiveness of Supervised Fine-Tuning (SFT) in acquiring new abilities through delta parameters
- Introduction of novel technique called DARE (Drop And REscale) to efficiently set most delta parameters to zero without compromising performance
- Demonstration of how DARE enables merger of multiple task-specific LMs into one model with diverse abilities
- Experimental evaluations on eight datasets show significant results, highlighting promising avenues for advancing natural language processing tasks efficiently

## Summary

, , , , 

In their paper titled "Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch," authors Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li explore how Language Models (LMs) can enhance their capabilities through parameter assimilation from similar models without the need for retraining or additional GPUs. The study focuses on encoder- or decoder-based LMs and highlights the effectiveness of Supervised Fine-Tuning (SFT) in acquiring new abilities through delta parameters. To efficiently set most delta parameters to zero without compromising performance, the researchers introduce a novel technique called DARE (Drop And REscale). They also demonstrate how DARE enables the merger of multiple task-specific LMs into one model with diverse abilities. Experimental evaluations on eight datasets show significant results, highlighting promising avenues for advancing natural language processing tasks efficiently.