# LLMMaps -- A Visual Metaphor for Stratified Evaluation of Large Language Models

**Authors:** Patrik Puchert, Poonam Poonam, Christian van Onzenoodt, Timo Ropinski  
**arXiv:** [2304.00457](https://arxiv.org/abs/2304.00457)  
**Date:** 2023-04-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.00457v1/

## Key points

- Large Language Models (LLMs) are impressive in various natural language processing tasks
- LLMs are prone to hallucinations and require diligent evaluation approaches
- Evaluations based on Q&A datasets for specific knowledge fields often report only a single accuracy number for the entire field, which is problematic for transparency and model improvement
- Ulm University researchers propose LLMMaps as a novel visualization technique to evaluate LLMs' performance with respect to Q&A datasets
- LLMMaps provide detailed insights into LLMs' knowledge capabilities in different subfields by transforming Q&A datasets and LLM responses into an internal knowledge structure
- Comparative visualization allows for the detailed comparison of multiple LLMs, displaying accuracy, number of questions answered correctly, and other relevant information
- Researchers conducted a comparative analysis of several state-of-the-art LLMs using LLMMaps and received positive feedback from participants who found the visualization unique and novel
- The proposed approach provides a stratified evaluation of one or several LLMs on one or several Q&A datasets - providing detailed insights into subfields where hallucinations are more likely to occur
- The approach can help better assess risks associated with using these models while guiding their further development.
- All necessary source code and data for generating LLMMaps will be available on GitHub.

## Summary

Large Language Models (LLMs) have revolutionized natural language processing, demonstrating impressive capabilities in various tasks such as language modeling, sentiment analysis, question-answering, and machine translation. However, these models are prone to hallucinations where they expose incorrect or false information in their responses. This renders diligent evaluation approaches mandatory. While LLM performance is often evaluated based on question and answer (Q&A) datasets for specific knowledge fields, such evaluations usually report only a single accuracy number for the entire field. This procedure is problematic with respect to transparency and model improvement. To address this issue, researchers at Ulm University propose LLMMaps as a novel visualization technique that enables users to evaluate LLMs' performance with respect to Q&A datasets. LLMMaps provide detailed insights into LLMs' knowledge capabilities in different subfields by transforming Q&A datasets as well as LLM responses into an internal knowledge structure. An extension for comparative visualization allows for the detailed comparison of multiple LLMs. In addition to displaying accuracy and the number of questions answered correctly, LLMMaps can also visualize further information relevant for the evaluation of LLMs. For example, textual legends can be integrated to display global information such as which model(s) has/have been evaluated on which dataset(s), and the overall score of the visualized models. To assess the effectiveness of LLMMaps, researchers conducted a comparative analysis of several state-of-the-art LLMs including BLOOM, GPT-2, GPT-3, ChatGPT and LLaMa-13B using them. They also received positive feedback from participants who found the visualization unique and novel. The proposed approach provides a stratified evaluation of one or several LLMs on one or several Q&A datasets - providing detailed insights into subfields where hallucinations are more likely to occur. The approach can help better assess risks associated with using these models while guiding their further development. All necessary source code and data for generating LLMMaps will be available on GitHub.