# Revealing the structure of language model capabilities

**Authors:** Ryan Burnell, Han Hao, Andrew R. A. Conway, Jose Hernandez Orallo  
**arXiv:** [2306.10062](https://arxiv.org/abs/2306.10062)  
**Date:** 2023-06-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.10062v1/

## Key points

- Authors aim to understand the capabilities of large language models (LLMs)
- Analyzed data from 29 different LLMs across 27 cognitive tasks
- Three distinct factors explain LLM capabilities: reasoning, comprehension, and core language modeling
- These factors account for a significant proportion of model performance variance
- Each ability shows different relationships to model properties such as size and instruction tuning
- Benchmarks for evaluating LLMs should focus on tasks that tap into each broad model ability
- Findings contribute to theoretical understanding of LLM capabilities and provide insights into their structure and relationships with model properties
- Implications for improving LLM design and evaluation methodologies

## Summary

In their study titled "Revealing the Structure of Language Model Capabilities," authors Ryan Burnell, Han Hao, Andrew R. A. Conway, and Jose Hernandez Orallo aim to understand the capabilities of large language models (LLMs) in order to predict and explain their behavior. They investigate the structure of LLM capabilities by analyzing data from 29 different LLMs across 27 cognitive tasks. Using a combination of Bayesian and frequentist factor analysis, the researchers extract latent capabilities from patterns of individual differences among the LLMs. They find that LLM capabilities are not monolithic but can be better explained by three distinct factors: reasoning, comprehension, and core language modeling. These three factors account for a significant proportion of the variance in model performance. The study reveals a consistent structure in the capabilities of different LLMs and highlights the multifaceted nature of these capabilities. Additionally, the authors observe that each of the three abilities shows different relationships to model properties such as size and instruction tuning. This finding helps refine our understanding of scaling laws and suggests that changes made to improve one ability may simultaneously impair others. Based on their findings, the authors propose that benchmarks for evaluating LLMs should focus on tasks that tap into each broad model ability. This approach could streamline benchmarking processes and provide a more comprehensive assessment of an LLM's overall performance. Overall, this study contributes to building a theoretical understanding of LLM capabilities and provides valuable insights into their structure and relationships with various model properties. The findings have implications for improving LLM design and evaluation methodologies which could help guide future research in this area.