# ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models

**Authors:** Aparna Elangovan, Ling Liu, Lei Xu, Sravan Bodapati, Dan Roth  
**arXiv:** [2405.18638](https://arxiv.org/abs/2405.18638)  
**Date:** 2024-05-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.18638v1/

## Key points

- Human evaluation of generative large language models (LLMs) should be approached as a multidisciplinary endeavor
- Factors such as usability, aesthetics, and cognitive biases are important in evaluating LLMs
- Effective test sets are crucial to differentiate between capabilities and weaknesses of powerful LLMs
- Cognitive biases can impact evaluations by conflating fluent information with truthfulness and affecting rating scores like Likert scales
- The framework ConSiDERS-The-Human evaluation consists of six pillars: Consistency, Scoring Criteria, Differentiating, User Experience, Responsible Evaluation Practices, and Scalability

## Summary

In this position paper, the authors argue that human evaluation of generative large language models (LLMs) should be approached as a multidisciplinary endeavor. They emphasize the importance of considering factors such as usability, aesthetics, and cognitive biases in evaluating LLMs to ensure reliable experimental design and results. The authors stress the need for effective test sets to differentiate between the capabilities and weaknesses of increasingly powerful LLMs. They also highlight how cognitive biases can impact evaluations by conflating fluent information with truthfulness and how cognitive uncertainty can affect rating scores like Likert scales. To accurately measure a model's capabilities, they address the critical role of test sets and propose a framework called ConSiDERS-The-Human evaluation framework consisting of six pillars: Consistency, Scoring Criteria, Differentiating, User Experience, Responsible Evaluation Practices, and Scalability. This paper contributes to rethinking human evaluation for generative large language models by advocating for a more comprehensive approach that considers various disciplines and factors to ensure accurate and reliable assessments in the age of advanced AI technologies.