# News Summarization and Evaluation in the Era of GPT-3

**Authors:** Tanya Goyal, Junyi Jessy Li, Greg Durrett  
**arXiv:** [2209.12356](https://arxiv.org/abs/2209.12356)  
**Date:** 2022-09-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2209.12356v2/

## Key points

- Recent advancements in large language models like GPT-3 have revolutionized NLP research
- The impact of GPT-3 on text summarization, specifically news summarization, is explored in this study
- GPT-3 summaries prompted using only a task description are preferred by humans and do not suffer from common dataset-specific issues such as poor factuality
- An A/B test was designed to collect preference annotations from human annotators for three summarization systems - BRIO, T0, and GPT3-D2
- Both reference-based and reference-free automatic metrics cannot reliably evaluate GPT-3 summaries
- The study evaluated models beyond generic summarization by focusing on keyword-based summarization and compared dominant fine-tuning approaches to prompting
- The authors released a corpus of 10K generated summaries from fine-tuned and prompt-based models across four standard summarization benchmarks along with 1K human preference judgments comparing different systems for generic and keyword-based summarization.
- This study highlights the potential of prompt-based GPT-3 models in generating high quality news summaries without suffering from common dataset specific issues while also highlighting the limitations of current evaluation metrics in assessing their performance accurately.

## Summary

The recent advancements in large language models like GPT-3 have revolutionized the field of natural language processing (NLP) research. In this study, the impact of GPT-3 on text summarization is explored, with a focus on news summarization as a classic benchmark domain. The study investigates how GPT-3 compares against fine-tuned models trained on large summarization datasets and shows that not only do humans overwhelmingly prefer GPT-3 summaries prompted using only a task description, but these also do not suffer from common dataset-specific issues such as poor factuality. To evaluate the performance of different summarization systems, an A/B test was designed to collect preference annotations from human annotators. The annotators were shown summaries from three summarization systems - BRIO, T0, and GPT3-D2 - for each given article and asked to select their most and least preferred summary or summaries based on their own preferences as non-expert consumers of news summaries. The study found that both reference-based and reference-free automatic metrics cannot reliably evaluate GPT-3 summaries. Additionally, the study evaluated models beyond generic summarization by focusing on keyword-based summarization and compared dominant fine-tuning approaches to prompting. To support further research in this area, the authors released a corpus of 10K generated summaries from fine-tuned and prompt-based models across four standard summarization benchmarks along with 1K human preference judgments comparing different systems for generic and keyword-based summarization. Overall, this study highlights the potential of prompt-based GPT-3 models in generating high quality news summaries without suffering from common dataset specific issues while also highlighting the limitations of current evaluation metrics in assessing their performance accurately.