# GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoning

**Authors:** Costas Mavromatis, George Karypis  
**arXiv:** [2405.20139](https://arxiv.org/abs/2405.20139)  
**Date:** 2024-05-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.20139v1/

## Key points

- Large Language Models (LLMs) are state-of-the-art models in Natural Language Processing (NLP) due to their ability to understand natural language but struggle with adapting to new or domain-specific information.
- Knowledge Graphs (KGs) store structured factual knowledge in triplets, capturing complex relationships between entities.
- Graph Neural Networks (GNNs) are effective for Question Answering over Knowledge Graphs (KGQA) tasks due to their ability to handle intricate graph structures.
- GNN-RAG combines LLMs' language understanding capabilities with GNNs' reasoning abilities in a retrieval-augmented generation style for KGQA tasks.
- GNN-RAG uses a GNN to reason over a dense subgraph of the KG, retrieves answer candidates, and utilizes shortest paths connecting question entities and answer candidates for reasoning within the graph.
- The framework leverages both GNNs and LLMs effectively, achieving state-of-the-art performance on popular KGQA benchmarks like WebQSP and CWQ, particularly excelling on multi-hop and multi-entity questions.

## Summary

, , , , 

In the field of Natural Language Processing (NLP), Large Language Models (LLMs) have emerged as state-of-the-art models due to their exceptional ability to understand natural language. These models are trained on vast amounts of textual data to acquire general human knowledge, but they struggle with adapting to new or domain-specific information and are prone to generating incorrect information. On the other hand, Knowledge Graphs (KGs) store structured factual knowledge in the form of triplets, forming a graph that captures complex relationships between entities. Question Answering over Knowledge Graphs (KGQA) involves answering questions by leveraging the information stored in KGs. Graph Neural Networks (GNNs) have been widely used for KGQA tasks because they can effectively handle the intricate graph structures present in KGs. In this study, a novel approach called GNN-RAG is introduced, which combines the language understanding capabilities of LLMs with the reasoning abilities of GNNs in a retrieval-augmented generation style. The GNN-RAG method first uses a GNN to reason over a dense subgraph of the KG and retrieve potential answer candidates for a given question. It then extracts shortest paths in the KG connecting question entities and answer candidates to represent reasoning paths within the graph. These paths are verbalized and fed into an LLM for further reasoning using retrieval-augmented generation techniques. The framework leverages the strengths of both GNNs and LLMs: GNN acts as a dense subgraph reasoner to extract valuable graph information, while LLM utilizes its natural language processing capabilities for effective KGQA. Additionally, a retrieval augmentation technique is developed to enhance performance further. Experimental results demonstrate that GNN-RAG achieves state-of-the-art performance on popular KGQA benchmarks such as WebQSP and CWQ, outperforming or matching even advanced models like GPT-4 when fine-tuned with 7B parameters. Notably, GNN-RAG excels particularly on multi-hop and multi-entity questions, surpassing existing approaches by significant margins at answer F1 scores. Overall, this study presents a comprehensive approach that effectively combines graph neural networks with large language models for improved Question Answering over Knowledge Graphs, showcasing superior performance on challenging QA tasks involving complex graph structures and multiple entities.