# Instruction Tuning with GPT-4

**Authors:** Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, Jianfeng Gao  
**arXiv:** [2304.03277](https://arxiv.org/abs/2304.03277)  
**Date:** 2023-04-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.03277v1/

## Key points

- Large language models (LLMs) are increasingly popular in natural language processing tasks
- Finetuning LLMs using machine-generated instruction-following data can enable remarkable zero-shot capabilities on new tasks without requiring human-written instructions
- GPT-4 is used to generate instruction-following data for LLM finetuning, leading to superior zero-shot performance on new tasks compared to previous state-of-the-art models
- GPT-4 tends to generate longer sequences than its predecessor, GPT 3.5
- Feedback and comparison data from GPT 4 is collected and used to train reward models for further evaluation of the approach
- The availability of open source models like BLOOM, GPT J, GPT NEO, OPT and LLaMA provides opportunities for researchers to develop more accessible and ethical AI systems aligned with human values.

## Summary

In recent years, the use of large language models (LLMs) has become increasingly popular in natural language processing tasks. Prior work has shown that finetuning LLMs using machine-generated instruction-following data can enable these models to achieve remarkable zero-shot capabilities on new tasks without requiring any human-written instructions. In this paper, the authors present the first attempt to use GPT-4, a state-of-the-art language model, to generate instruction-following data for LLM finetuning. The authors conduct early experiments on instruction-tuned LLaMA models and show that the 52K English and Chinese instruction-following data generated by GPT-4 leads to superior zero-shot performance on new tasks compared to previous state-of-the art models. The frequency distributions of sequence length are also compared between GPT-4 and GPT 3.5, with GPT 4 tending to generate longer sequences than its predecessor. To further evaluate their approach, the authors collect feedback and comparison data from GPT 4 and use it to train reward models. They make their data generated using GPT 4 as well as their codebase publicly available. The paper also discusses open source efforts towards developing general purpose text based assistants aligned with human values. Early attempts include BLOOM, GPT J, GPT NEO, OPT and LLaMA which have drawn significant interest in promoting work towards creating more accessible and ethical AI systems. Overall, this paper presents a promising approach for improving zero shot performance on new tasks through instruction tuning LLMs using machine generated data from state of the art language models like GPT 4. The availability of open source models also provides opportunities for researchers to develop more accessible and ethical AI systems aligned with human values.