# PersonaGym: Evaluating Persona Agents and LLMs

**Authors:** Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik Narasimhan, Vishvak Murahari  
**arXiv:** [2407.18416](https://arxiv.org/abs/2407.18416)  
**Date:** 2024-07-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2407.18416v1/

## Key points

- PersonaGym introduced as a dynamic evaluation framework for assessing persona agents in large language models (LLMs)
- Model complexity does not necessarily lead to improved persona agent abilities, highlighting the critical role of PersonaGym in evaluating these agents effectively
- PersonaScore established as an automated metric based on decision theory to quantify persona agent capabilities across five key evaluation tasks
- Evaluation tasks categorized into Normative Evaluation, Prescriptive Evaluation, and Descriptive Evaluation based on decision theory principles
- Importance emphasized of accurately and comprehensively assessing persona agents' performance using innovative frameworks like PersonaGym and PersonaScore

## Summary

, , , , 

In this study, the researchers introduce PersonaGym, a dynamic evaluation framework designed to assess persona agents in large language models (LLMs). The findings highlight that model complexity does not necessarily equate to improved persona agent abilities. This emphasizes the critical role of PersonaGym in effectively evaluating these agents. Additionally, the researchers establish PersonaScore as an automated metric, based on decision theory, to quantify the capabilities of persona agents across five key evaluation tasks. These tasks are grounded in decision theory principles and provide a comprehensive assessment of persona agent performance. The study benchmarks the PersonaScore of 200 persona agents using six open and closed-source LLMs on a dataset comprising 10,000 agent-relevant questions. The evaluation tasks are categorized into Normative Evaluation, Prescriptive Evaluation, and Descriptive Evaluation based on decision theory principles. Normative Evaluation focuses on optimal decision-making in given environments, while Prescriptive Evaluation involves prescribing how agents should act based on linguistic habits, persona consistency, and toxicity control. Descriptive Evaluation aims to understand why agents make specific decisions through tasks like Action Justification. Overall, the research sheds light on the importance of accurately and comprehensively assessing persona agents' performance. By utilizing decision theory principles and introducing innovative evaluation frameworks like PersonaGym and PersonaScore, the study provides valuable insights into enhancing persona agent capabilities across various applications. The findings also underscore the need for continued algorithmic and architectural advancements to develop more faithful and performant persona agents in LLMs.