Can ChatGPT Assess Human Personalities? A General Evaluation Framework

Haocong Rao; Cyril Leung; Chunyan Miao

AI summary · arXiv: 2303.01248v2 · cs.CL

Can ChatGPT Assess Human Personalities? A General Evaluation Framework

Haocong Rao, Cyril Leung, Chunyan Miao

Original paper Submitted to arXiv on 01 Mar. 2023 Our codes are available at https://github.com/Kali-Hac/ChatGPT-MBTI License: NONEXCLUSIVE-DISTRIB 1.0

How to read this summary? Illustrated guide

The essentials

Large Language Models (LLMs) like ChatGPT have potential to exhibit human-like psychology
Existing works have studied virtual personalities of LLMs but not analyzed human personalities via LLMs
Authors present a generic evaluation framework for LLMs to assess human personalities based on Myers Briggs Type Indicator (MBTI) tests
Framework involves devising unbiased prompts by randomly permuting options in MBTI questions and adopting the average testing result to encourage more impartial answer generation
The authors propose replacing the subject in question statements to enable flexible queries and assessments on different subjects from LLMs
They re-formulate the question instructions in a manner of correctness evaluation to facilitate LLMs to generate clearer responses.
This framework enables LLMs to flexibly assess personalities of different groups of people.
Three evaluation metrics proposed: consistency, robustness, and fairness of assessment results from state-of-the-art LLMs including ChatGPT and InstructGPT.
Experiments reveal ChatGPT's ability to assess human personalities with consistent and fairer assessments despite lower robustness against prompt biases compared with InstructGPT.
This study opens up new avenues for exploring how AI can better understand human psychology.

Key points written by AI: they may contain errors, check them against the paper.

AI-generated keywords: Large Language Models

Read further

Comprehensive Summary
Layman's Summary
Blog article
Original abstract

The potential of Large Language Models (LLMs), particularly ChatGPT, to exhibit human-like psychology is still largely unexplored. While existing works have studied the virtual personalities of LLMs, they rarely explore the possibility of analyzing human personalities via LLMs. In this paper, authors Haocong Rao, Cyril Leung, and Chunyan Miao present a generic evaluation framework for LLMs to assess human personalities based on Myers Briggs Type Indicator (MBTI) tests. The proposed framework involves devising unbiased prompts by randomly permuting options in MBTI questions and adopting the average testing result to encourage more impartial answer generation. The authors also propose replacing the subject in question statements to enable flexible queries and assessments on different subjects from LLMs. Finally, they re-formulate the question instructions in a manner of correctness evaluation to facilitate LLMs to generate clearer responses. This framework enables LLMs to flexibly assess personalities of different groups of people. The authors further propose three evaluation metrics to measure the consistency, robustness, and fairness of assessment results from state-of-the-art LLMs including ChatGPT and InstructGPT. Their experiments reveal ChatGPT's ability to assess human personalities with consistent and fairer assessments despite lower robustness against prompt biases compared with InstructGPT. Overall, this paper presents an innovative approach that allows for more nuanced analysis of human personality traits using advanced language models like ChatGPT. By providing a comprehensive evaluation framework that measures consistency, robustness, and fairness in assessing different groups' personalities through MBTI tests, this study opens up new avenues for exploring how AI can better understand human psychology.

Exploring the Potential of Large Language Models to Assess Human Personality Traits

In recent years, large language models (LLMs) such as ChatGPT have become increasingly popular for their ability to generate human-like conversations. However, the potential of these LLMs to exhibit human-like psychology is still largely unexplored. While existing works have studied the virtual personalities of LLMs, they rarely explore the possibility of analyzing human personalities via LLMs.

The proposed framework involves devising unbiased prompts by randomly permuting options in MBTI questions and adopting the average testing result to encourage more impartial answer generation. The authors also propose replacing the subject in question statements to enable flexible queries and assessments on different subjects from LLMs. Finally, they re-formulate the question instructions in a manner of correctness evaluation to facilitate LLMs to generate clearer responses.

This framework enables LLMs to flexibly assess personalities of different groups of people. To measure this capability further, the authors propose three evaluation metrics: consistency which measures how well an assessment result matches with its ground truth; robustness which evaluates how well an assessment result resists prompt biases; and fairness which measures whether an assessment result is biased towards certain groups or not.

The authors conducted experiments using state-of-the-art LLM models including ChatGPT and InstructGPT on two datasets: one containing MBTI test results from real people and another containing simulated results generated by humans following MBTI guidelines. Their experiments revealed ChatGPT's ability to assess human personalities with consistent and fairer assessments despite lower robustness against prompt biases compared with InstructGPT.

Conclusion

Overall, this paper presents an innovative approach that allows for more nuanced analysis of human personality traits using advanced language models like ChatGPT.

Created on 18 Mar. 2023

Was this synthesis useful?

Score: 0

Ask questions about this paper to our AI assistant

How to chat with this paper? Illustrated guide

Compare with the PDF

You can also chat with multiple papers at once here.

AI assistant instructions?

Tools for this paper

Explain it differently 3 free rewrites left today

Save to a collection Group papers in collections to compare them, chat with them and export them. Create a free account Log in

Export BibTeX · RIS (Zotero, Mendeley, EndNote) · Markdown (Notion, Obsidian)

Advanced tools Premium

For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium

Zotero

Citation alerts Citation alerts guide

Who works on it? The people and institutions who publish on its topic

Journal club Premium+ Slides to present it

Versions What changed between versions

Talk about it in public Premium+ Press kit: release, questions, hype check Press kit guide

Papers around this one

See them on a map

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

Very close
PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits
Hang Jiang, Xiajie Zhang, Xubo Cao et al. · 2023 · Computation and Language · AI summary available

Abstract
Despite the many use cases for large language models (LLMs) in creating personalized chatbots, there has been limited research on evaluating the extent to which the behaviors of personalized LLMs accurately and consistently reflect specific personality traits. We consider studying the behavior of LLM-based agents which we refer to as LLM personas and present a case study with GPT-3.5 and GPT-4 to investigate whether LLMs can generate content that aligns with their assigned personality profiles. To this end, we simulate distinct LLM personas based on the Big Five personality model, have them complete the 44-item Big Five Inventory (BFI) personality test and a story writing task, and then asse…
Very close
Personality Traits in Large Language Models
Greg Serapio-García, Mustafa Safdari, Clément Crepy et al. · 2023 · Computation and Language · Not summarized yet

Abstract
The advent of large language models (LLMs) has revolutionized natural language processing, enabling the generation of coherent and contextually relevant human-like text. As LLMs increasingly power conversational agents used by the general public world-wide, the synthetic personality embedded in these models, by virtue of training on large amounts of human data, is becoming increasingly important. Since personality is a key factor determining the effectiveness of communication, we present a comprehensive method for administering and validating personality tests on widely-used LLMs, as well as for shaping personality in the generated text of such LLMs. Applying this method, we found: 1) person…
Very close
Can ChatGPT and Bard Generate Aligned Assessment Items? A Reliability Analysis against Human Performance
Abdolvahab Khademi · 2023 · Journal of Applied Learning and Teaching 2023 · Not summarized yet

Abstract
ChatGPT and Bard are AI chatbots based on Large Language Models (LLM) that are slated to promise different applications in diverse areas. In education, these AI technologies have been tested for applications in assessment and teaching. In assessment, AI has long been used in automated essay scoring and automated item generation. One psychometric property that these tools must have to assist or replace humans in assessment is high reliability in terms of agreement between AI scores and human raters. In this paper, we measure the reliability of OpenAI ChatGP and Google Bard LLMs tools against experienced and trained humans in perceiving and rating the complexity of writing prompts. Intraclass …
Very close
BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues
Haodong Duan, Jueqi Wei, Chonghua Wang et al. · 2023 · Computation and Language · AI summary available

Abstract
Interacting with human via high-quality multi-turn dialogues is a key feature of large language models (LLMs). However, human-based evaluation of such capability involves intensive manual labor. This report provides a preliminary evaluation of existing large language models for human-style multi-turn chatting, through an LLM-based approach. We start from real-world human dialogues and keep the very first utterances as the ChatSEED. Then we prompt LLMs to generate a full multi-turn dialogue (tens of utterances) based on the ChatSEED, utterance by utterance. Finally, we adopt state-of-the-art LLMs (GPT-4, \etc) as the judge to evaluate the generated dialogues. With different evaluation protoco…
Very close
LLM Agents in Interaction: Measuring Personality Consistency and Linguistic Alignment in Interacting Populations of Large Language Models
Ivar Frisch, Mario Giulianelli · 2024 · Computation and Language · AI summary available

Abstract
While both agent interaction and personalisation are vibrant topics in research on large language models (LLMs), there has been limited focus on the effect of language interaction on the behaviour of persona-conditioned LLM agents. Such an endeavour is important to ensure that agents remain consistent to their assigned traits yet are able to engage in open, naturalistic dialogues. In our experiments, we condition GPT-3.5 on personality profiles through prompting and create a two-group population of LLM agents using a simple variability-inducing sampling algorithm. We then administer personality tests and submit the agents to a collaborative writing task, finding that different profiles exhib…
Very close
ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning
Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh et al. · 2023 · Computation and Language · AI summary available

Abstract
Over the last few years, large language models (LLMs) have emerged as the most important breakthroughs in natural language processing (NLP) that fundamentally transform research and developments in the field. ChatGPT represents one of the most exciting LLM systems developed recently to showcase impressive skills for language generation and highly attract public attention. Among various exciting applications discovered for ChatGPT in English, the model can process and generate texts for multiple languages due to its multilingual training data. Given the broad adoption of ChatGPT for English in different problems and areas, a natural question is whether ChatGPT can also be applied effectively …
Very close
PersonaGym: Evaluating Persona Agents and LLMs
Vinay Samuel, Henry Peng Zou, Yue Zhou et al. · 2024 · Computation and Language · AI summary available

Abstract
Persona agents, which are LLM agents that act according to an assigned persona, have demonstrated impressive contextual response capabilities across various applications. These persona agents offer significant enhancements across diverse sectors, such as education, healthcare, and entertainment, where model developers can align agent responses to different user requirements thereby broadening the scope of agent applications. However, evaluating persona agent performance is incredibly challenging due to the complexity of assessing persona adherence in free-form interactions across various environments that are relevant to each persona agent. We introduce PersonaGym, the first dynamic evaluati…
Very close
A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee et al. · 2023 · Computation and Language · AI summary available

Abstract
This paper proposes a framework for quantitatively evaluating interactive LLMs such as ChatGPT using publicly available data sets. We carry out an extensive technical evaluation of ChatGPT using 23 data sets covering 8 different common NLP application tasks. We evaluate the multitask, multilingual and multi-modal aspects of ChatGPT based on these data sets and a newly designed multimodal dataset. We find that ChatGPT outperforms LLMs with zero-shot learning on most tasks and even outperforms fine-tuned models on some tasks. We find that it is better at understanding non-Latin script languages than generating them. It is able to generate multimodal content from textual prompts, via an interme…

Go further

How are these papers found?

We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.

For now only among the papers summarized on this site: search again to add the prior work, the follow-ups and Semantic Scholar’s recommendations.

The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.