# Podcast Summary Assessment: A Resource for Evaluating Summary Assessment Methods

**Authors:** Potsawee Manakul, Mark J. F. Gales  
**arXiv:** [2208.13265](https://arxiv.org/abs/2208.13265)  
**Date:** 2022-08-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2208.13265v1/

## Key points

- Automatic summary assessment is valuable for evaluating machine-generated and human-produced summaries.
- It helps in developing summary generation systems and detecting inappropriate summaries.
- Summary assessment can be done by ranking summary generation systems, ranking specific document summaries, or estimating the quality of a document-summary pair on an absolute scale.
- Existing datasets for summary assessment are typically based on news summarization datasets like CNN/DailyMail or XSum.
- The podcast summary assessment corpus is a new dataset that contains long-input speech-based documents and was evaluated by human experts at TREC2020.
- This dataset is unique as it allows identification of inappropriate reference summaries within the podcast corpus.
- The authors explore existing assessment methods including model-free and model-based approaches and present benchmark results using this dataset.
- Summary assessment is also applied for data selection to filter reference summary-document pairings for training purposes.
- The experimental results provide insights into both the summary assessment and generation tasks.
- The podcast summary assessment data is publicly available.

## Summary

Automatic summary assessment is a valuable tool for evaluating both machine-generated and human-produced summaries. It allows for the development of summary generation systems and helps detect inappropriate summaries. Summary assessment can be conducted in various modes such as ranking summary generation systems, ranking summaries of a specific document or estimating the quality of a document-summary pair on an absolute scale. Existing datasets for summary assessment are typically based on news summarization datasets like CNN/DailyMail or XSum. This work introduces a new dataset called the podcast summary assessment corpus which consists of podcast summaries that were evaluated by human experts at TREC2020. This dataset is unique as it contains long-input speech-based documents and provides an opportunity to identify inappropriate reference summaries within the podcast corpus. The authors explore existing assessment methods including model-free and model-based approaches and present benchmark results using this long-input summary assessment dataset. Additionally, they apply summary assessment for data selection to filter reference summary-document pairings for training purposes. The experimental results shed light on both the summary assessment and generation tasks. The podcast summary assessment data is publicly available.