# OpenChat: Advancing Open-source Language Models with Mixed-Quality Data

**Authors:** Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, Yang Liu  
**arXiv:** [2309.11235](https://arxiv.org/abs/2309.11235)  
**Date:** 2023-09-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.11235v1/

## Key points

- Open-source large language models like LLaMA have gained popularity
- Models have been improved through supervised fine-tuning (SFT) and reinforcement learning fine-tuning (RLFT)
- SFT treats all training data equally, while RLFT requires high-quality preference data
- OpenChat is a framework proposed to advance open-source language models using mixed-quality data
- OpenChat focuses on SFT training data with a small amount of expert data and a large proportion of sub-optimal data without preference labels
- OpenChat introduces C(onditioned)-RLFT, which treats different data sources as reward labels and learns a class-conditioned policy
- C(onditioned)-RLFT can be solved through single-stage, RL-free supervised learning, avoiding costly human preference labeling
- Extensive experiments were conducted on three standard benchmarks to evaluate the performance of OpenChat.

## Summary

Open-source large language models like LLaMA have become increasingly popular in recent years. These models have been enhanced through supervised fine-tuning (SFT) and reinforcement learning fine-tuning (RLFT) techniques to align them with human goals. However, SFT methods treat all training data with mixed quality equally, while RLFT methods require high-quality pairwise or ranking-based preference data. To address these limitations, the authors of this study propose a novel framework called OpenChat. The goal of OpenChat is to advance open-source language models using mixed-quality data. Specifically, they focus on the general SFT training data, which consists of a small amount of expert data mixed with a large proportion of sub-optimal data without any preference labels. The authors introduce a method called C(onditioned)-RLFT as part of the OpenChat framework. This approach treats different data sources as coarse-grained reward labels and learns a class-conditioned policy to leverage complementary information about the quality of the data. Interestingly, the optimal policy in C-RLFT can be easily solved through single-stage, RL-free supervised learning, which is lightweight and avoids the need for costly human preference labeling. Extensive experiments were conducted on three standard benchmarks to evaluate the performance of OpenChat.