# Towards Multi-Modal DBMSs for Seamless Querying of Texts and Tables

**Authors:** Matthias Urban, Carsten Binnig  
**arXiv:** [2304.13559](https://arxiv.org/abs/2304.13559)  
**Date:** 2023-04-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.13559v1/

## Key points

- Multi-Modal Databases (MMDBs) integrate text collections as tables within traditional relational database systems
- Key innovation: Use of multi-modal operators (MMOps) based on advanced language models like GPT-3
- Architecture components include Multi-modal Database Storage, MMDB-Model, and multi-modal SQL queries
- MMDB-Model extracts structured data from text collections based on specified schema for queryable attributes
- Query efficiency enhanced by multi-modal materialized views and indexes
- Experimental evaluations show MMDB prototype outperforms existing approaches in accuracy and performance with less training data required
- Optimizations explored to address efficiency challenges within the MMDB system
- Contribution towards advancing Multi-Modal DBMSs for seamless querying of textual data and traditional tables

## Summary

Multi-Modal Databases (MMDBs) are a new type of database system that allows for seamless querying of both text and tables using SQL. This is made possible by integrating text collections as tables within a traditional relational database system. The key innovation of MMDBs lies in the use of multi-modal operators (MMOps) based on advanced language models like GPT-3. These MMOps allow text collections to be treated as tables without the need for manual data transformation. The architecture of an MMDB consists of several components. The Multi-modal Database Storage component enables the integration of text collections as tables by specifying only the schema of queryable attributes. This is facilitated by the MMDB-Model, which learns to extract structured data from text collections based on the specified schema. Users can then issue multi-modal SQL queries that are translated into query plans containing traditional and multi-modal database operators such as joins and scans. The MMDB-Model computes representations of query attributes and texts to generate output table data by extracting values from the text. To enhance query efficiency in an MMDB environment, multi-modal materialized views and indexes are also available. In experimental evaluations, it has been shown that our MMDB prototype outperforms existing approaches in terms of accuracy and performance while requiring less training data for model fine-tuning. Additionally, optimizations have been explored to address efficiency challenges within the MMDB system. In conclusion, this paper contributes towards advancing Multi-Modal DBMSs for seamless querying of both textual data and traditional tables. By offering a unified database framework for handling diverse types of information, MMDBs provide a promising solution for modern database systems.