# Measuring and Narrowing the Compositionality Gap in Language Models

**Authors:** Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, Mike Lewis  
**arXiv:** [2210.03350](https://arxiv.org/abs/2210.03350)  
**Date:** 2022-10-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2210.03350v1/

## Key points

- Authors investigate language models' ability to perform compositional reasoning tasks
- Introduce metric called compositionality gap to measure model's ability to generate overall solution
- Design multi-hop questions to evaluate metric, requiring composition of multiple facts
- Analyze GPT-3 models and find that single-hop performance improves faster than multi-hop performance as model size increases
- Propose elicitive prompting techniques like "chain of thought" to narrow compositionality gap
- Introduce "self-ask" method where model asks itself follow-up questions before answering initial question
- Self-ask approach improves upon chain of thought by structuring prompting process
- Structured prompting allows easy integration with search engine for answering follow-up questions and enhancing accuracy in compositional reasoning tasks
- Research highlights limitations and potential solutions for improving language models' compositional reasoning ability.

## Summary

In the paper titled "Measuring and Narrowing the Compositionality Gap in Language Models," authors Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis investigate the ability of language models to perform compositional reasoning tasks. They focus on tasks where the overall solution depends on correctly combining answers to sub-problems. The authors introduce a metric called the compositionality gap which measures how often models can answer all sub-problems correctly but fail to generate the overall solution. To evaluate this metric, the authors design multi-hop questions that require composing multiple facts unlikely to have been observed together during pretraining. They specifically analyze models from the GPT-3 family and find that as model size increases single-hop question answering performance improves faster than multi-hop performance. This observation suggests that while more powerful models can memorize and recall more factual knowledge their ability to perform compositional reasoning does not improve proportionally. To address this issue, the authors propose elicitive prompting techniques such as "chain of thought" to narrow the compositionality gap. These techniques encourage explicit reasoning instead of relying solely on implicit understanding. Additionally they introduce a new method called "self-ask," where the model asks itself follow-up questions before answering the initial question. This approach further improves upon chain of thought by structuring the prompting process. Furthermore, the authors demonstrate that self-ask's structured prompting allows for easy integration with a search engine to answer follow-up questions enhancing accuracy in compositional reasoning tasks. Overall this research sheds light on both limitations and potential solutions for improving language models' ability to perform compositional reasoning effectively.