# SAGE: A Realistic Benchmark for Semantic Understanding

**Authors:** Samarth Goel, Reagan J. Lee, Kannan Ramchandran  
**arXiv:** [2509.21310](https://arxiv.org/abs/2509.21310)  
**Date:** 2025-09-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2509.21310v1/

## Key points

- Large language models (LLMs) excel on traditional benchmarks but require more challenging evaluation frameworks for deeper semantic understanding.
- The Semantic Alignment & Generalization Evaluation (SAGE) benchmark assesses embedding models and similarity metrics across five key categories: Human Preference Alignment, Transformation Robustness, Information Sensitivity, Clustering Performance, and Retrieval Robustness.
- SAGE includes tasks to evaluate the ability of similarity metrics to distinguish between superficial changes that preserve meaning and semantic alterations that fundamentally alter content.
- Another critical task in SAGE focuses on detecting and quantifying semantic degradation as content is modified by inserting irrelevant content or removing content spans from long-form datasets.
- SAGE also includes tasks like Clustering Performance and Retrieval Robustness to assess how well similarity metrics preserve categorical structure in unsupervised settings and handle text corruptions in real-world retrieval systems.
- Through comprehensive evaluations across multiple datasets using different perturbations and transformations, SAGE uncovers significant performance gaps among embedding models and classical metrics.
- SAGE exposes critical limitations in current semantic understanding capabilities while providing a more realistic assessment of model robustness for real-world deployment.

## Summary

In the realm of large language models (LLMs), the need for more challenging evaluation frameworks that delve deeper into semantic understanding has become increasingly urgent as these models continue to excel on traditional benchmarks. To address this gap, the Semantic Alignment & Generalization Evaluation (SAGE) benchmark was introduced. <br>
 is a rigorous benchmark designed to assess both embedding models and similarity metrics across five key categories: Human Preference Alignment, Transformation Robustness, Information Sensitivity, Clustering Performance, and Retrieval Robustness. One of the tasks within  is , which aims to evaluate the ability of similarity metrics to distinguish between superficial changes that preserve meaning and semantic alterations that fundamentally alter content. This task involves applying various transformations to long-form text datasets such as academic papers, legislation, and news articles, including superficial perturbations like typos or synonym replacements and semantic changes like negation or factual modifications. The goal is to measure how well a metric can differentiate between original-to-superficial changes, original-to-semantic changes, and original-to-summary similarities. Another critical task in  is , which focuses on whether similarity metrics can accurately detect and quantify semantic degradation as content is modified. By inserting irrelevant content or removing content spans from long-form datasets, this task evaluates how well metrics track changes in meaning proportionally with the amount of perturbation applied. Furthermore,< kd>SAGE</ kd > includes tasks such as < kd > Clustering Performance</ kd >and< kd > Retrieval Robustness</ kd >to assess how well similarity metrics preserve categorical structure in unsupervised settings and handle text corruptions in real-world retrieval systems respectively. Through comprehensive evaluations across multiple datasets using different perturbations and transformations, < kd > SAGE </kd> uncovers significant performance gaps among embedding models and classical metrics. Overall, < kd > SAGE </kd> exposes critical limitations in current semantic understanding capabilities while providing a more realistic assessment of model robustness for real-world deployment. The benchmark highlights the need for continued research and development in creating more advanced evaluation frameworks that push the boundaries of semantic understanding in language models.