# Evaluating Large Language Models in Semantic Parsing for Conversational Question Answering over Knowledge Graphs

**Authors:** Phillip Schneider, Manuel Klettner, Kristiina Jokinen, Elena Simperl, Florian Matthes  
**arXiv:** [2401.01711](https://arxiv.org/abs/2401.01711)  
**Date:** 2024-01-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.01711v1/

## Key points

- The paper explores the use of semantic parsing in conversational question answering systems
- The study focuses on knowledge-based conversational question answering using graph queries
- Large language models are evaluated for their performance in generating graph queries
- Experiments were conducted using a benchmark dataset and models of varying sizes
- Common issue types in the generated output were identified
- Few-shot prompting and fine-tuning techniques showed significant improvements, especially for smaller models
- The research contributes to understanding the effectiveness of large language models in semantic parsing for conversational question answering over knowledge graphs
- Insights are provided on optimizing model performance and addressing challenges in generating accurate and contextually relevant graph queries.

## Summary

This paper titled "Evaluating Large Language Models in Semantic Parsing for Conversational Question Answering over Knowledge Graphs" explores the use of semantic parsing in conversational question answering systems. These systems rely on generating structured database queries from natural language inputs to enable interactive information retrieval. Specifically, the study focuses on knowledge-based conversational question answering, where dialogue utterances are transformed into graph queries to retrieve facts stored within a knowledge graph. The authors evaluate the performance of large language models that have not been explicitly pre-trained on this task. They conduct a series of experiments using an extensive benchmark dataset and compare models of varying sizes with different prompting techniques. Additionally, they identify common issue types in the generated output. The results demonstrate that large language models are capable of generating graph queries from dialogues. The study highlights significant improvements achievable through few-shot prompting and fine-tuning techniques, particularly for smaller models that exhibit lower zero-shot performance. Overall, this research contributes to understanding the effectiveness of large language models in semantic parsing for conversational question answering over knowledge graphs. The findings provide insights into optimizing model performance and addressing challenges associated with generating accurate and contextually relevant graph queries in information-seeking conversations.