# Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study

**Authors:** Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, Dongmei Zhang  
**arXiv:** [2305.13062](https://arxiv.org/abs/2305.13062)  
**Date:** 2023-05-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.13062v5/

## Key points

- Large language models (LLMs) are increasingly used as few-shot reasoners for Natural Language (NL)-related tasks
- LLMs' ability to process structured data like tables is relatively unexplored
- A benchmark has been introduced to evaluate LLMs' structural understanding capabilities through tasks such as cell lookup, row retrieval, and size detection
- Performance of advanced LLM models like GPT-3.5 and GPT-4 varies based on input choices including table input format, content order, role prompting, and partition marks
- "Self-augmentation" method proposed for effective structural prompting using internal knowledge within LLMs leads to significant improvements in performance on tabular tasks
- Different table storage formats (CSV, JSON, XML, markdown, HTML) impact LLM comprehension abilities
- Accurate partitioning of data is important for downstream tasks involving tabular datasets paired with external knowledge sources

## Summary

Large language models (LLMs) are increasingly being utilized as few-shot reasoners for Natural Language (NL)-related tasks. However, their ability to process structured data like tables remains a relatively unexplored area. While tables can be serialized as input for LLMs, there is a lack of comprehensive studies on whether LLMs truly comprehend this type of data. In an effort to address this gap, this paper introduces a benchmark designed to evaluate the structural understanding capabilities of LLMs through seven distinct tasks such as cell lookup, row retrieval, and size detection. The study focuses on evaluating the performance of advanced LLM models, specifically GPT-3.5 and GPT-4, across different input choices including table input format, content order, role prompting, and partition marks. The results reveal that the performance of LLMs varies based on these input choices. Drawing from insights gained through benchmark evaluations, the paper proposes a method called "self-augmentation" for effective structural prompting using internal knowledge within LLMs. By combining self-augmentation with carefully selected input choices, significant improvements in LLM performance on various tabular tasks such as TabFact (+2.31%), HybridQA (+2.13%), SQA (+2.72%), Feverous (+0.84%), and ToTTo (+5.68%) are observed. The open-source benchmark and proposed prompting methods presented in this study offer valuable insights for future research in this domain. Additionally, the paper delves into the impact of different table storage formats (CSV, JSON, XML, markdown, HTML) on LLM comprehension abilities and explores the importance of accurate partitioning of data for downstream tasks involving tabular datasets paired with external knowledge sources. Overall,this study sheds light on the potential of large language models in understanding structured data like tables and provides valuable guidance for optimizing their performance in processing tabular information effectively.