# GreaseLM: Graph REASoning Enhanced Language Models for Question Answering

**Authors:** Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D. Manning, Jure Leskovec  
**arXiv:** [2201.08860](https://arxiv.org/abs/2201.08860)  
**Date:** 2022-01-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2201.08860v1/

## Key points

- Question answering requires reasoning over explicit constraints and unstated relevant knowledge about the world
- Pretrained language models (LM) are dominant in natural language processing for question answering tasks but do not robustly represent latent relationships between concepts necessary for reasoning
- Knowledge graphs (KG) are used to augment LMs with structured representations of world knowledge, but effectively fusing and reasoning over KG representations and language context remains an open question
- GreaseLM is a new model that fuses encoded representations from pretrained LMs and graph neural networks (GNN) over multiple layers of modality interaction operations
- GreaseLM allows language context representations to be grounded by structured world knowledge and linguistic nuances such as negation and hedging in the context to inform the graph representations of knowledge
- Results on three benchmarks demonstrate that GreaseLM can more reliably answer questions that require reasoning over both situational constraints and structured knowledge, even outperforming larger models.
- Various methods have been proposed to enhance LM performance using KGs, but they still face challenges in effectively integrating contextual information with structured knowledge representation in KGs.
- Overall, GreaseLM represents a promising approach for enhancing LM performance by effectively fusing contextual information with structured world knowledge representation through GNNs.

## Summary

The task of question answering requires reasoning over both explicit constraints described in the textual context and unstated relevant knowledge about the world. Pretrained language models (LM) have become the dominant paradigm in natural language processing for question answering tasks, but they do not robustly represent latent relationships between concepts necessary for reasoning. Knowledge graphs (KG) are often used to augment LMs with structured representations of world knowledge, but it remains an open question how to effectively fuse and reason over KG representations and language context, which provides situational constraints and nuances. To address this issue, a new model called GreaseLM has been proposed. GreaseLM fuses encoded representations from pretrained LMs and graph neural networks (GNN) over multiple layers of modality interaction operations. Information from both modalities propagates to the other, allowing language context representations to be grounded by structured world knowledge, and allowing linguistic nuances such as negation and hedging in the context to inform the graph representations of knowledge. The results on three benchmarks in commonsense reasoning (CommonsenseQA, OpenbookQA) and medical question answering (MedQA-USMLE) domains demonstrate that GreaseLM can more reliably answer questions that require reasoning over both situational constraints and structured knowledge. It even outperforms models eight times larger than itself. Various methods have been proposed to enhance LM performance using KGs, including strong fine-tuning techniques, multi-task learning approaches or hybrid models combining LMs with external KGs. However, these methods still face challenges in effectively integrating contextual information with structured knowledge representation in KGs. Overall, GreaseLM represents a promising approach for enhancing LM performance by effectively fusing contextual information with structured world knowledge representation through GNNs.