# WebGPT: Browser-assisted question-answering with human feedback

**Authors:** Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, John Schulman  
**arXiv:** [2112.09332](https://arxiv.org/abs/2112.09332)  
**Date:** 2021-12-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2112.09332v1/

## Key points

- The paper presents a novel approach to fine-tune GPT-3 for answering long-form questions using a text-based web-browsing environment.
- The model is able to search and navigate the web, making it more effective in providing accurate answers.
- Humans can perform the task, enabling them to train models on the task using imitation learning and optimize answer quality with human feedback.
- Models must collect references while browsing in support of their answers to make human evaluation of factual accuracy easier.
- The authors' best model is obtained by fine-tuning GPT-3 using behavior cloning and then performing rejection sampling against a reward model trained to predict human preferences.
- The authors' model's answers are preferred by humans 56% of the time compared to those provided by their human demonstrators and 69% of the time compared to the highest-voted answer from Reddit.
- This innovative approach combines machine learning techniques with human feedback to achieve high performance levels.
- It provides insights into how such algorithms can be used for natural language processing (NLP) and question-answering systems.

## Summary

The paper titled "WebGPT: Browser-assisted question-answering with human feedback" presents a novel approach to fine-tune GPT-3 for answering long-form questions using a text-based web-browsing environment. This allows the model to search and navigate the web, making it more effective in providing accurate answers. The authors set up the task in such a way that it can be performed by humans, enabling them to train models on the task using imitation learning and optimize answer quality with human feedback. To make human evaluation of factual accuracy easier, models must collect references while browsing in support of their answers. The authors train and evaluate their models on ELI5, a dataset of questions asked by Reddit users. Their best model is obtained by fine-tuning GPT-3 using behavior cloning and then performing rejection sampling against a reward model trained to predict human preferences. The authors' model's answers are preferred by humans 56% of the time compared to those provided by their human demonstrators and 69% of the time compared to the highest-voted answer from Reddit. This innovative approach combines machine learning techniques with human feedback to achieve high performance levels. It provides insights into how such algorithms can be used for natural language processing (NLP) and question-answering systems. The study was conducted by Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe Tyna Eloundou Gretchen Krueger Kevin Button Matthew Knight Benjamin Chess John Schulman.