# Can ChatGPT Assess Human Personalities? A General Evaluation Framework

**Authors:** Haocong Rao, Cyril Leung, Chunyan Miao  
**arXiv:** [2303.01248](https://arxiv.org/abs/2303.01248)  
**Date:** 2023-03-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.01248v2/

## Key points

- Large Language Models (LLMs) like ChatGPT have potential to exhibit human-like psychology
- Existing works have studied virtual personalities of LLMs but not analyzed human personalities via LLMs
- Authors present a generic evaluation framework for LLMs to assess human personalities based on Myers Briggs Type Indicator (MBTI) tests
- Framework involves devising unbiased prompts by randomly permuting options in MBTI questions and adopting the average testing result to encourage more impartial answer generation
- The authors propose replacing the subject in question statements to enable flexible queries and assessments on different subjects from LLMs
- They re-formulate the question instructions in a manner of correctness evaluation to facilitate LLMs to generate clearer responses.
- This framework enables LLMs to flexibly assess personalities of different groups of people.
- Three evaluation metrics proposed: consistency, robustness, and fairness of assessment results from state-of-the-art LLMs including ChatGPT and InstructGPT.
- Experiments reveal ChatGPT's ability to assess human personalities with consistent and fairer assessments despite lower robustness against prompt biases compared with InstructGPT.
- This study opens up new avenues for exploring how AI can better understand human psychology.

## Summary

The potential of Large Language Models (LLMs), particularly ChatGPT, to exhibit human-like psychology is still largely unexplored. While existing works have studied the virtual personalities of LLMs, they rarely explore the possibility of analyzing human personalities via LLMs. In this paper, authors Haocong Rao, Cyril Leung, and Chunyan Miao present a generic evaluation framework for LLMs to assess human personalities based on Myers Briggs Type Indicator (MBTI) tests. The proposed framework involves devising unbiased prompts by randomly permuting options in MBTI questions and adopting the average testing result to encourage more impartial answer generation. The authors also propose replacing the subject in question statements to enable flexible queries and assessments on different subjects from LLMs. Finally, they re-formulate the question instructions in a manner of correctness evaluation to facilitate LLMs to generate clearer responses. This framework enables LLMs to flexibly assess personalities of different groups of people. The authors further propose three evaluation metrics to measure the consistency, robustness, and fairness of assessment results from state-of-the-art LLMs including ChatGPT and InstructGPT. Their experiments reveal ChatGPT's ability to assess human personalities with consistent and fairer assessments despite lower robustness against prompt biases compared with InstructGPT. Overall, this paper presents an innovative approach that allows for more nuanced analysis of human personality traits using advanced language models like ChatGPT. By providing a comprehensive evaluation framework that measures consistency, robustness, and fairness in assessing different groups' personalities through MBTI tests, this study opens up new avenues for exploring how AI can better understand human psychology.