# Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training

**Authors:** Oshin Agarwal, Heming Ge, Siamak Shakeri, Rami Al-Rfou  
**arXiv:** [2010.12688](https://arxiv.org/abs/2010.12688)  
**Date:** 2020-10-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2010.12688v2/

## Key points

- The paper discusses converting knowledge graph (KG) triples into natural text, specifically focusing on verbalizing the entire English Wikidata KG.
- The authors highlight the unique challenges of verbalizing a broad, open-domain, large-scale KG like Wikidata.
- The proposed approach converts the KG into natural text to enable seamless integration into existing language models.
- Converting KGs into natural text offers advantages such as improved factual accuracy and reduced toxicity in language models.
- Evaluation of the approach shows significant improvements in knowledge-intensive tasks like open domain question answering (QA) and the LAMA knowledge probe.
- Prior work in Data-To-Text Generation focused on domain-specific benchmark datasets, while this study tackles verbalizing an encyclopedic KG like Wikidata.
- The research contributes to bridging the gap between structured KGs and natural language corpora.
- The paper presents a novel approach to generating synthetic corpora by converting KG triples from English Wikidata into natural text.
- Results demonstrate the effectiveness of this approach in enhancing language models' performance while ensuring improved factual accuracy and reduced toxicity.

## Summary

This paper titled "Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training" discusses the task of converting knowledge graph (KG) triples into natural text, specifically focusing on verbalizing the entire English Wikidata KG. The authors highlight the unique challenges associated with this broad, open-domain, large-scale verbalization and demonstrate how verbalizing a comprehensive KG like Wikidata can facilitate the integration of structured KGs and natural language corpora. Unlike existing architectures that aim to integrate these two sources, the proposed approach converts the KG into natural text, enabling seamless integration into existing language models. This conversion offers several advantages, including improved factual accuracy and reduced toxicity in the resulting language model. To evaluate this approach, the authors augment the retrieval corpus in a retrieval language model and observe significant improvements in knowledge-intensive tasks such as open domain question answering (QA) and the LAMA knowledge probe. The authors emphasize that prior work in Data-To-Text Generation primarily focused on domain-specific benchmark datasets. In contrast, their study tackles the challenge of verbalizing an encyclopedic KG like Wikidata. By expanding on this research area, they contribute to bridging the gap between structured KGs and natural language corpora. Overall, this paper presents a novel approach to generating synthetic corpora by converting KG triples into natural text from English Wikidata. The results demonstrate its effectiveness in enhancing language models' performance on knowledge-intensive tasks while ensuring improved factual accuracy and reduced toxicity.