# Benchmarking Large Language Models for News Summarization

**Authors:** Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, Tatsunori B. Hashimoto  
**arXiv:** [2301.13848](https://arxiv.org/abs/2301.13848)  
**Date:** 2023-01-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2301.13848v1/

## Key points

- Large language models (LLMs) have potential in automatic summarization
- Human evaluation conducted on 10 LLMs across different pretraining methods, prompts, and model scales
- Instruction tuning is key to LLM's zero-shot summarization capability rather than model size
- Existing studies limited by low-quality references leading to underestimates of human performance and lower few-shot and fine-tuning performance
- High-quality summaries collected from freelance writers for human evaluation
- Quality of freelance writer summaries evaluated using Mechanical Turkers found to be much higher than original reference summaries in CNN/DM and XSUM
- Little difference between quality of freelance writer summaries and those generated by Instruct Davinci model
- LLM-generated summaries and freelance writer-generated summaries had distinctive styles with regard to paraphrasing and copying from source articles
- Annotators recruited to compare Instruct Davinci-generated summaries with those written by freelance writers; overall, Instruct Davinci rated as comparable to freelance writers but individual annotators showed varying preferences
- Instruction tuning crucial for LLMs' summarization capability rather than model scale
- Issues with low-quality references used in previous studies highlighted
- Proposed collecting better quality summaries from freelance writers as a solution
- Findings contribute towards improving LLM evaluation techniques for future research in automatic summarization.

## Summary

Large language models (LLMs) have shown great potential in automatic summarization, but the reasons behind their success are not well understood. To address this issue, a human evaluation was conducted on ten LLMs across different pretraining methods, prompts, and model scales. The study made two important observations: first, instruction tuning is the key to the LLM's zero-shot summarization capability rather than model size; second, existing studies have been limited by low-quality references leading to underestimates of human performance and lower few-shot and fine-tuning performance. To better evaluate LLMs, high-quality summaries were collected from freelance writers for human evaluation. To ensure consistency in summary length between the freelance writer summaries and those generated by Instruct Davinci model, a new prompt was introduced that elicited summaries around 50 words long. The quality of the freelance writer summaries was evaluated using Mechanical Turkers and found to be much higher than the original reference summaries in CNN/DM and XSUM. Additionally, there was little difference between the quality of the freelance writer summaries and those generated by Instruct Davinci. Despite similar performance in quality control studies, LLM-generated summaries and freelance writer-generated summaries had distinctive styles with regard to paraphrasing and copying from source articles. Extractiveness measures were used to compare coverage and density between these two types of summaries. Annotators were recruited to compare Instruct Davinci-generated summaries with those written by freelance writers. On aggregate, Instruct Davinci was rated as comparable to freelance writers; however, individual annotators showed varying preferences for either Instruct Davinci or freelance writers. Overall, this study identified instruction tuning as crucial for LLMs' summarization capability rather than model scale. It also highlighted issues with low-quality references used in previous studies and proposed collecting better quality summaries from freelance writers as a solution. These findings contribute towards improving LLM evaluation techniques for future research in automatic summarization.