# Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models

**Authors:** Guillermo Ortiz-Jimenez, Alessandro Favero, Pascal Frossard  
**arXiv:** [2305.12827](https://arxiv.org/abs/2305.12827)  
**Date:** 2023-05-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.12827v1/

## Key points

- Task arithmetic is a method for editing pre-trained models in weight space
- Combining fine-tuned weights of different tasks can enhance model performance on those tasks
- Removing these weights can lead to task forgetting
- Weight disentanglement is crucial for making task arithmetic effective
- Fine-tuning models in their tangent space through linearization amplifies weight disentanglement
- This leads to significant performance improvements across various task arithmetic benchmarks and diverse models
- There is a link between task arithmetic and the spatial localization of NTK eigenfunctions, as revealed by theoretical and empirical analyses of neural tangent kernel (NTK)
- These findings offer a more reliable and effective approach to editing pre-trained models by leveraging NTK linearization

## Summary

In recent years, task arithmetic has emerged as a promising method for editing pre-trained models in weight space. By combining the fine-tuned weights of different tasks, this approach can enhance the model's performance on those tasks. Conversely, removing these weights can lead to task forgetting. However, our understanding of the underlying principles and effectiveness of task arithmetic is still limited. To address this gap, we conducted a comprehensive study focusing on vision-language models. Our research revealed that weight disentanglement plays a crucial role in making task arithmetic effective. This property arises during the pre-training phase and becomes evident when distinct directions in weight space govern separate localized regions in function space associated with specific tasks. We also discovered that fine-tuning models in their tangent space through linearization amplifies weight disentanglement. This finding resulted in significant performance improvements across various task arithmetic benchmarks and diverse models. Building upon these insights, we conducted theoretical and empirical analyses of the neural tangent kernel (NTK) of these models. Through this analysis, we established a compelling link between task arithmetic and the spatial localization of NTK eigenfunctions. Overall, our work provides novel insights into the fundamental mechanisms of task arithmetic and offers a more reliable and effective approach to editing pre-trained models by leveraging NTK linearization. These findings have important implications for improving model performance and advancing the field of vision-language modeling.