# Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization

**Authors:** Yixin Liu, Alexander R. Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, Arman Cohan  
**arXiv:** [2311.09184](https://arxiv.org/abs/2311.09184)  
**Date:** 2023-11-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.09184v2/

## Key points

- Study title: "Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization"
- Researchers: Yixin Liu, Alexander R. Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, Arman Cohan
- Purpose: Explore performance of large language models (LLMs) in instruction controllable text summarization
- Curated an evaluation-only dataset with source article and natural language requirement for desired summary characteristics
- Human evaluations of five LLM-based systems for instruction-following capabilities
- Benchmarking LLM-based automatic evaluation using four protocols and 11 LLMs (40 evaluation methods)
- Instruction controllable text summarization remains challenging for LLMs with factual errors and other mistakes in summaries
- None of the LLM-based evaluation methods achieved strong alignment with human annotators
- Significant performance gaps among different LLMs in both summary generation and evaluation capabilities
- GitHub repository: https://github.com/yale-nlp/InstruSum
- LLM-evaluators Leaderboard: https://huggingface.co/spaces/yale-nlp/InstruSumEval

## Summary

In their study titled "Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization," authors Yixin Liu, Alexander R. Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, and Arman Cohan explore the performance of large language models (LLMs) in instruction controllable text summarization. The researchers curated an evaluation-only dataset for this specific task setting where the model input includes both a source article and a natural language requirement for desired summary characteristics. <br>
The study involved human evaluations of five LLM-based systems to assess their instruction-following capabilities in controllable summarization. Additionally, the researchers benchmarked LLM-based automatic evaluation using four different evaluation protocols and 11 LLMs, resulting in 40 evaluation methods. <br>
The findings revealed that instruction controllable text summarization remains a challenging task for LLMs as all evaluated models still made factual errors and other types of mistakes in their summaries. Furthermore, none of the LLM-based evaluation methods were able to achieve strong alignment with human annotators when assessing the quality of candidate summaries. The study also highlighted significant performance gaps among different LLMs in both summary generation and evaluation capabilities.<br>
To support future research in this area,. This research was presented at  and additional resources including a GitHub repository (https://github.com/yale-nlp/InstruSum) and an LLM-evaluators Leaderboard (https://huggingface.co/spaces/yale-nlp/InstruSumEval) were provided for further exploration by interested parties.