# Holistic Evaluation of Language Models

**Authors:** Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman  
**arXiv:** [2211.09110](https://arxiv.org/abs/2211.09110)  
**Date:** 2022-11-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2211.09110v1/

## Key points

- Language Models (LMs) are crucial for a wide range of applications in the evolving landscape of language technologies.
- There is a lack of comprehensive understanding regarding the capabilities, limitations, and risks associated with LMs.
- The Holistic Evaluation of Language Models (HELM) framework aims to address this knowledge gap and enhance transparency within the field.
- HELM categorizes scenarios and metrics relevant to LMs to identify areas for improvement and gaps that need addressing.
- HELM utilizes a multi-metric approach encompassing seven key metrics: accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency.
- HELM evaluates language models across 42 scenarios, significantly improving evaluation coverage from an average of 17.9% to 96%.
- Results from HELM's evaluation process provide valuable insights into the strengths and weaknesses of different language models.
- Raw model prompts and completions are publicly released alongside a general modular toolkit to promote transparency within the research community.
- HELM is envisioned as a dynamic benchmark that will evolve over time with new scenarios, metric enhancements, and inclusion of emerging language models.

## Summary

In the rapidly evolving landscape of language technologies, Language Models (LMs) have emerged as the cornerstone for a wide range of applications. However, despite their increasing prevalence and importance, there remains a lack of comprehensive understanding regarding the capabilities, limitations, and potential risks associated with these models. To address this gap in knowledge and enhance transparency within the field, a team of researchers has introduced the concept of Holistic Evaluation of Language Models (HELM). The HELM framework is designed to provide a structured approach to evaluating language models by first categorizing the diverse array of scenarios and metrics relevant to LMs. By systematically organizing this vast space of use cases and desired outcomes, HELM aims to identify key areas where improvements can be made and gaps that need to be addressed. This initial phase involves selecting a subset of scenarios based on their coverage and feasibility while also highlighting any overlooked or underrepresented aspects such as question answering for marginalized English dialects or metrics related to trustworthiness. One distinctive feature of HELM is its multi-metric approach, which goes beyond traditional accuracy measurements to encompass seven key metrics: accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency. These metrics are applied across 16 core scenarios whenever possible to ensure a comprehensive evaluation that sheds light on various dimensions of model performance. Additionally, targeted evaluations are conducted for specific aspects like reasoning and disinformation using 26 focused scenarios. In a groundbreaking large-scale evaluation involving 30 prominent language models – including open-source, limited-access, and closed models – HELM assesses these models across all 42 scenarios identified in the framework. Notably, 21 of these scenarios were previously unexplored in mainstream LM evaluations. By benchmarking all models on standardized conditions using consistent metrics across core scenarios,
HELM significantly improves the overall evaluation coverage from an average of 17.9% to an impressive 96%. The results generated through HELM's rigorous evaluation process yield 25 top-level findings that offer valuable insights into the strengths and weaknesses of different language models. To promote transparency and facilitate further analysis within the research community, all raw model prompts and completions are publicly released alongside a general modular toolkit. Moving forward, HELM is envisioned as a dynamic benchmark that will evolve over time with new scenarios, metrics enhancements,and inclusion of emerging language models to ensure ongoing relevance in assessing language technologies comprehensively.