MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Close Cites this paper Uses this paper in depth Not relevant

   Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Framework

   E. Arias, Hannah Blocher, Julian Rodemann et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 10 times · Not summarized yet

   Abstract

   Open-ended text generation has become a prominent task in natural language processing due to the rise of powerful (large) language models. However, evaluating the quality of these models and the employed decoding strategies remains challenging due to trade-offs among widely used metrics such as coherence, diversity, and perplexity. This paper addresses the specific problem of multicriteria evaluation for open-ended text generation, proposing novel methods for both relative and absolute rankings of decoding methods. Specifically, we employ benchmarking approaches based on partial orderings and present a new summary metric to balance existing automatic indicators, providing a more holistic eva…

2. Close Not relevant

   Distributional Discrepancy: A Metric for Unconditional Text Generation

   Ping Cai, Xing-Yuan Chen, Peng Jin et al. · 2020 · Knowledge-Based Systems · cited 7 times · Not summarized yet

   Abstract

   The purpose of unconditional text generation is to train a model with real sentences, then generate novel sentences of the same quality and diversity as the training data. However, when different metrics are used for comparing the methods of unconditional text generation, contradictory conclusions are drawn. The difficulty is that both the diversity and quality of the sample should be considered simultaneously when the models are evaluated. To solve this problem, a novel metric of distributional discrepancy (DD) is designed to evaluate generators based on the discrepancy between the generated and real training sentences. However, it cannot compute the DD directly because the distribution of …

3. Close Not relevant

   Open-Domain Text Evaluation via Contrastive Distribution Methods

   Sidi Lu, Hongyi Liu, Asli Celikyilmaz et al. · 2023 · International Conference on Machine Learning · cited 2 times · Not summarized yet

   Abstract

   Recent advancements in open-domain text generation, driven by the power of large pre-trained language models (LLMs), have demonstrated remarkable performance. However, assessing these models' generation quality remains a challenge. In this paper, we introduce a novel method for evaluating open-domain text generation called Contrastive Distribution Methods (CDM). Leveraging the connection between increasing model parameters and enhanced LLM performance, CDM creates a mapping from the _contrast_ of two probabilistic distributions -- one known to be superior to the other -- to quality measures. We investigate CDM for open-domain text generation evaluation under two paradigms: 1) _Generative_ CD…

4. Close Cited by this paper A method it uses Not relevant

   MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance

   Wei Zhao, Maxime Peyrard, Fei Liu et al. · 2019 · Conference on Empirical Methods in Natural Language Processing · cited 757 times · Not summarized yet

   Abstract

   A robust evaluation metric has a profound impact on the development of text generation systems. A desirable metric compares system output against references based on their semantics rather than surface forms. In this paper we investigate strategies to encode system and reference texts to devise a metric that shows a high correlation with human judgment of text quality. We validate our new metric, namely MoverScore, on a number of text generation tasks including summarization, machine translation, image captioning, and data-to-text generation, where the outputs are produced by a variety of neural and non-neural systems. Our findings suggest that metrics combining contextualized representation…

5. Close Not relevant

   Language Model Evaluation in Open-ended Text Generation

   An Nguyen · 2021 · arXiv.org · preprint, not peer reviewed · cited 3 times · Not summarized yet

   Abstract

   Although current state-of-the-art language models have achieved impressive results in numerous natural language processing tasks, still they could not solve the problem of producing repetitive, dull and sometimes inconsistent text in open-ended text generation. Studies often attribute this problem to the maximum likelihood training objective, and propose alternative approaches by using stochastic decoding methods or altering the training objective. However, there is still a lack of consistent evaluation metrics to directly compare the efficacy of these solutions. In this work, we study different evaluation metrics that have been proposed to evaluate quality, diversity and consistency of mach…

6. Close Cited by this paper Not relevant

   UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation

   Jian Guan, Minlie Huang · 2020 · Conference on Empirical Methods in Natural Language Processing · cited 82 times · Not summarized yet

   Abstract

   Despite the success of existing referenced metrics (e.g., BLEU and MoverScore), they correlate poorly with human judgments for open-ended text generation including story or dialog generation because of the notorious one-to-many issue: there are many plausible outputs for the same input, which may differ substantially in literal or semantics from the limited number of given references. To alleviate this issue, we propose UNION, a learnable unreferenced metric for evaluating open-ended story generation, which measures the quality of a generated story without any reference. Built on top of BERT, UNION is trained to distinguish human-written stories from negative samples and recover the perturba…

7. Close Cites this paper Uses this paper in depth Not relevant

   SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics

   Gregor Kornhardt, Moritz Piening, J. Chemseddine et al. · 2026 · cited 1 time · Not summarized yet

   Abstract

   Evaluating text generation requires measuring how well the generated distribution matches the data distribution. For autoregressive models, this is done by the perplexity. Diffusion and flow-based language models can only provide a likelihood bound, whose tightness differs between model families. Sample-based substitutes such as generative perplexity with entropy do not consider the distribution fit. We propose SOL, a distance between text distributions. Each sequence is represented by the empirical measure of its hidden states under a fixed transformer and the distributions of these measures are compared by the double sliced Wasserstein distance. We prove that SOL is a metric if the transfo…

8. Close Cites this paper Uses this paper in depth Not relevant

   Reference-Based Analysis of Coherence and Diversity in Open-Ended Text Generation

   E. Arias · 2026 · Not summarized yet

   Abstract

   Evaluating open-ended text generation involves understanding how different properties of a continuation relate to its perceived quality. We present a reference-based framework for examining coherence and diversity through three perspectives: aligning their evolution with human trajectories, comparing their summaries with a human continuation of the same prompt, and estimating their likelihood under a human reference distribution. Experiments with human quality ratings suggest that diversity-based alignment and mean-based comparisons capture quality-related variation, although the comparisons do not establish a predictive advantage for temporal alignment over simpler baselines. Reference like…

Prior work it builds on

1. Related Used in depth by this paper A method it uses Not relevant

   The Curious Case of Neural Text Degeneration

   Ari Holtzman, Jan Buys, Li Du et al. · 2019 · International Conference on Learning Representations · cited 4,736 times · Not summarized yet

   Abstract

   Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, using likelihood as a decoding objective leads to text that is bland and strangely repetitive. In this paper, we reveal surprising distributional differences between human text and machine text. In addition, we find that decoding strategies alone can dramatically effect the quality of machine text, even when generated from exactly the same neura…

2. Related Used in depth by this paper A method it uses Not relevant

   Sparse Text Generation

   Pedro Henrique Martins, Zita Marinho, André F. T. Martins · 2020 · Conference on Empirical Methods in Natural Language Processing · cited 46 times · Not summarized yet

   Abstract

   Current state-of-the-art text generators build on powerful language models such as GPT-2, achieving impressive performance. However, to avoid degenerate text, they require sampling from a modified softmax, via temperature parameters or ad-hoc truncation techniques, as in top-$k$ or nucleus sampling. This creates a mismatch between training and testing conditions. In this paper, we use the recently introduced entmax transformation to train and sample from a natively sparse language model, avoiding this mismatch. The result is a text generator with favorable performance in terms of fluency and consistency, fewer repetitions, and n-gram diversity closer to human text. In order to evaluate our m…

3. Close A method it uses Not relevant

   Automatic Detection of Generated Text is Easiest when Humans are Fooled

   Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch et al. · 2019 · Annual Meeting of the Association for Computational Linguistics · cited 571 times · Not summarized yet

   Abstract

   Recent advancements in neural language modelling make it possible to rapidly generate vast amounts of human-sounding text. The capabilities of humans and automatic discriminators to detect machine-generated text have been a large source of research interest, but humans and machines rely on different cues to make their decisions. Here, we perform careful benchmarking and analysis of three popular sampling-based decoding strategies—top-_k_, nucleus sampling, and untruncated random sampling—and show that improvements in decoding methods have primarily optimized for fooling humans. This comes at the expense of introducing statistical abnormalities that make detection easy for automatic systems. …

4. Close Not relevant

   Evaluation of Text Generation: A Survey

   Asli Celikyilmaz, Elizabeth Clark, Jian-Feng Gao · 2020 · arXiv.org · preprint, not peer reviewed · cited 503 times · Not summarized yet

   Abstract

   The paper surveys evaluation methods of natural language generation (NLG) systems that have been developed in the last few years. We group NLG evaluation methods into three categories: (1) human-centric evaluation metrics, (2) automatic metrics that require no training, and (3) machine-learned metrics. For each category, we discuss the progress that has been made and the challenges still being faced, with a focus on the evaluation of recently proposed NLG tasks and neural NLG models. We then present two case studies of automatic text summarization and long text generation, and conclude the paper by proposing future research directions.

5. Related Not relevant

   On Accurate Evaluation of GANs for Language Generation

   Stanislau Semeniuta, A. Severyn, S. Gelly · 2018 · arXiv.org · preprint, not peer reviewed · cited 87 times · Not summarized yet

   Abstract

   Generative Adversarial Networks (GANs) are a promising approach to language generation. The latest works introducing novel GAN models for language generation use n-gram based metrics for evaluation and only report single scores of the best run. In this paper, we argue that this often misrepresents the true picture and does not tell the full story, as GAN models can be extremely sensitive to the random initialization and small deviations from the best hyperparameter choice. In particular, we demonstrate that the previously used BLEU score is not sensitive to semantic deterioration of generated texts and propose alternative metrics that better capture the quality and diversity of the generated…

Papers that cite it

1. Close Uses this paper in depth Not relevant

   Coherence-Aware Distributional Evaluation of Open-Ended Text Generation

   Jin-Nuo Liu, Jun-Hao Zhu, Wei-Feng Jiang et al. · 2026 · Not summarized yet

   Abstract

   Existing open-ended generation metrics measure likelihood, lexical diversity, or distributional similarity in generic representation space, yet can miss fundamental dimensions of quality. A prominent blind spot is global coherence: a generated passage may be locally fluent while remaining globally contradictory, causally inconsistent, or topically disconnected. We identify representation as a central bottleneck in detecting these failures and introduce CHORD (Coherence-aware Hidden-state Open-generation Reference Distance), a coherence-sensitive distributional metric. CHORD encodes generated and human-written corpora in the hidden-state space of a frozen LLM using a coherence-eliciting promp…

2. Close Uses this paper in depth Not relevant

   MATCHA: Matching Text via Contrastive Semantic Alignment

   Siran Li, Ece Sena Etoglu, Carsten Eickhoff et al. · 2026 · Annual Meeting of the Association for Computational Linguistics · cited 1 time · Not summarized yet

   Abstract

   Reliable evaluation is essential for understanding large language model (LLM) performance, yet today's go-to metrics, namely token-overlap scores (e.g., ROUGE) and embedding-based measures (e.g., BERTScore), often misjudge semantic similarity of documents. Our study shows that both token-overlap metrics and embedding-based metrics routinely assign nearly identical scores to texts that directly contradict each other, thereby potentially masking fundamental errors. We introduce MATCHA, an automatic metric that jointly rewards semantic agreement with a reference and penalizes contradictions. MATCHA employs a dual-view perspective that measures (i) proximity to the gold text and (ii) distance fr…

3. Related Uses this paper in depth Not relevant

   Machine-generated text detection prevents language model collapse

   George Drayson, Emine Yilmaz, Vasileios Lampos · 2025 · Conference on Empirical Methods in Natural Language Processing · cited 22 times · Not summarized yet

   Abstract

   As Large Language Models (LLMs) become increasingly prevalent, their generated outputs are proliferating across the web, risking a future where machine-generated content dilutes human-authored text. Since online data is the primary resource for LLM pre-training, subsequent models could be trained on an unknown portion of synthetic samples. This could lead to model collapse, a degenerative process whereby LLMs reinforce their own errors, reduce output diversity, and ultimately yield declining performance. In this study, we investigate the impact of decoding strategy on model collapse, analysing the text characteristics at each model generation, the similarity to human references, and the resu…

4. Related Uses this paper in depth Not relevant

   SynthTextEval: Synthetic Text Data Generation and Evaluation for High-Stakes Domains

   Krithika Ramesh, Daniel Smolyak, Zihao Zhao et al. · 2025 · Conference on Empirical Methods in Natural Language Processing · cited 8 times · Not summarized yet

   Abstract

   We present SynthTextEval, a toolkit for conducting comprehensive evaluations of synthetic text. The fluency of large language model (LLM) outputs has made synthetic text potentially viable for numerous applications, such as reducing the risks of privacy violations in the development and deployment of AI systems in high-stakes domains. Realizing this potential, however, requires principled consistent evaluations of synthetic data across multiple dimensions: its utility in downstream systems, the fairness of these systems, the risk of privacy leakage, general distributional differences from the source text, and qualitative feedback from domain experts. SynthTextEval allows users to conduct eva…

5. Close Not relevant

   Aligning Text-to-Music Evaluation with Human Preferences

   Yichen Huang, Zachary Novack, Koichi Saito et al. · 2025 · International Society for Music Information Retrieval Conference · cited 24 times · Not summarized yet

   Abstract

   Despite significant recent advances in generative acoustic text-to-music (TTM) modeling, robust evaluation of these models lags behind, relying in particular on the popular Fr\'echet Audio Distance (FAD). In this work, we rigorously study the design space of reference-based divergence metrics for evaluating TTM models through (1) designing four synthetic meta-evaluations to measure sensitivity to particular musical desiderata, and (2) collecting and evaluating on MusicPrefs, the first open-source dataset of human preferences for TTM systems. We find that not only is the standard FAD setup inconsistent on both synthetic and human preference data, but that nearly all existing metrics fail to e…