# Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic

**Authors:** Rishabh Bhardwaj, Do Duc Anh, Soujanya Poria  
**arXiv:** [2402.11746](https://arxiv.org/abs/2402.11746)  
**Date:** 2024-02-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.11746v1/

## Key points

- Authors address the challenge of compromised safety in fine-tuned language models
- Introduce RESTA method to restore safety through task arithmetic by adding a safety vector to model weights
- Effectiveness of RESTA demonstrated in parameter-efficient and full fine-tuning scenarios across various tasks (instruction following, problem-solving) in Chinese, English, Hindi
- Generalizability of RESTA shown on existing safety evaluation benchmarks and multilingual benchmark dataset with harmful questions
- RESTA significantly reduces harmfulness of compromised models while maintaining task performance
- Source codes for RESTA available on GitHub
- Promising results for RESTA's impact on Chinese and Vietnamese languages
- Performance comparisons show improved task-specific scores with RESTA compared to base model Llama-2, SFT with dropout DARE, and their combination (RESTAd)
- Analysis on CATQA dataset reveals significant increases in unsafety scores for Llama-2 safe model under different evaluations
- Research provides insights into enhancing language model safety during fine-tuning processes using arithmetic methods like RESTA

## Summary

In their paper titled "Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic," authors Rishabh Bhardwaj, Do Duc Anh, and Soujanya Poria address the challenge faced by aligned language models where fine-tuning often compromises safety. They introduce a method called RESTA (REstoring Safety through Task Arithmetic) to overcome this limitation by adding a safety vector to the weights of the compromised model. The effectiveness of RESTA is demonstrated in both parameter-efficient and full fine-tuning scenarios across various downstream tasks such as instruction following in Chinese, English, and Hindi, as well as problem-solving in Code and Math. The study also showcases the generalizability of RESTA on existing safety evaluation benchmarks and introduces a multilingual benchmark dataset with harmful questions across different categories. Overall, RESTA significantly reduces the harmfulness of compromised models while maintaining task performance. Source codes for RESTA are provided on GitHub. Further evaluations on Chinese and Vietnamese languages show promising results for RESTA's impact. Performance comparisons between base model Llama-2, SFT with dropout DARE, added safety vector RESTA, and their combination (RESTAd) reveal improved task-specific performance scores. Additionally, analysis on versions of the safety evaluation dataset CATQA highlights significant increases in unsafety scores for the Llama-2 safe model when subjected to different evaluations. This research provides valuable insights into enhancing the safety of language models during fine-tuning processes and demonstrates the efficacy of using arithmetic methods like RESTA to realign model safety across diverse tasks and languages.