# Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models

**Authors:** Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, Quanquan Gu  
**arXiv:** [2401.01335](https://arxiv.org/abs/2401.01335)  
**Date:** 2024-01-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.01335v1/

## Key points

- Authors propose a new fine-tuning method called Self-Play Fine-Tuning (SPIN) to improve Large Language Models (LLMs)
- SPIN incorporates a self-play mechanism where LLM plays against instances of itself
- LLM generates its own training data from previous iterations and distinguishes it from human-annotated data
- The authors provide theoretical proof that the global optimum for their method is achieved when LLM policy aligns with target data distribution
- SPIN is evaluated on benchmark datasets and significantly improves LLM performance
- SPIN outperforms models trained through direct preference optimization (DPO) with extra GPT-4 preference data
- Self-play enables achieving human-level performance in LLMs without relying on expert opponents
- SPIN harnesses self-play to convert weak language models into strong ones

## Summary

In this paper titled "Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models," authors Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu propose a new fine-tuning method called Self-Play fIne-tuNing (SPIN) to improve the performance of Large Language Models (LLMs). The authors emphasize the importance of human-annotated data in advancing LLMs through Supervised Fine-Tuning (SFT). However, their aim is to enhance LLMs without acquiring additional human-annotated data. The SPIN method begins with a supervised fine-tuned model and incorporates a self-play mechanism at its core. The LLM refines its capabilities by playing against instances of itself. It generates its own training data from previous iterations and distinguishes these self-generated responses from those obtained from human-annotated data. This iterative process progressively elevates the LLM from a nascent model to a formidable one, unlocking the full potential of human-annotated demonstration data for SFT. The authors provide theoretical proof that the global optimum for the training objective function of their method is achieved only when the LLM policy aligns with the target data distribution. To validate their approach empirically, they evaluate SPIN on several benchmark datasets including the HuggingFace Open LLM Leaderboard, MT-Bench, and datasets from Big-Bench. The results demonstrate that SPIN significantly improves the performance of LLMs across various benchmarks. In fact, it even outperforms models trained through direct preference optimization (DPO) supplemented with extra GPT-4 preference data. This highlights the promise of self-play as it enables achieving human-level performance in LLMs without relying on expert opponents. Overall, this research introduces an innovative fine-tuning method that harnesses self-play to convert weak language models into strong ones. The findings suggest that SPIN can effectively leverage existing human-annotated data and enhance LLM performance, paving the way for further advancements in natural language processing.