# RAIN: Your Language Models Can Align Themselves without Finetuning

**Authors:** Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, Hongyang Zhang  
**arXiv:** [2309.07124](https://arxiv.org/abs/2309.07124)  
**Date:** 2023-09-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.07124v1/

## Key points

- Large language models (LLMs) have inconsistencies with human preferences in natural language processing
- Previous research attempted to align LLMs with human preferences using reinforcement learning or instruction tuning
- Aligning frozen LLMs without additional data is desirable
- The study proposes a method called Rewindable Auto-regressive INference (RAIN) to align unaligned LLMs directly without extra data
- RAIN integrates self-evaluation and rewind mechanisms for consistent responses with human preferences through self-boosting
- RAIN operates without training, gradient computation, or parameter updates
- During the self-evaluation phase, the model receives guidance on alignment through a fixed-template prompt
- Experimental results show the effectiveness of RAIN evaluated by GPT-4 and humans
- RAIN significantly improves harmlessness rate of LLaMA 30B from 82% to 97% on HH dataset while maintaining helpfulness rate
- RAIN establishes a new defense baseline against adversarial attacks llm-attacks on Vicuna 33B by reducing attack success rate from 94% to 19%
- This work presents a novel approach for pre-trained LLMs to align with human preferences without finetuning or additional data
- RAIN shows promising results in improving model behavior and defending against adversarial attacks.

## Summary

In the field of natural language processing, large language models (LLMs) have shown inconsistencies with human preferences. Previous research has attempted to align these models with human preferences by gathering preference data and using reinforcement learning or instruction tuning for fine-tuning. However, aligning frozen LLMs without any additional data is more desirable. This study explores the potential of aligning unaligned LLMs directly without the need for extra data. The researchers propose a method called Rewindable Auto-regressive INference (RAIN) that integrates self-evaluation and rewind mechanisms to enable unaligned LLMs to generate responses consistent with human preferences through self-boosting. RAIN operates without requiring extra data for model alignment and avoids training, gradient computation, or parameter updates. During the self-evaluation phase, the model receives guidance on which human preference to align with through a fixed-template prompt, eliminating the need to modify the initial prompt. Experimental results evaluated by GPT-4 and humans demonstrate the effectiveness of RAIN. On the HH dataset, RAIN significantly improves the harmlessness rate of LLaMA 30B over vanilla inference from 82% to 97%, while maintaining helpfulness rate. Additionally, under an adversarial attack called llm-attacks on Vicuna 33B, RAIN establishes a new defense baseline by reducing attack success rate from 94% to 19%. Overall, this work presents a novel approach that allows pre-trained LLMs to align themselves with human preferences without finetuning or additional data. The proposed RAIN method shows promising results in improving model behavior and defending against adversarial attacks.