# Structured information extraction from complex scientific text with fine-tuned large language models

**Authors:** Alexander Dunn, John Dagdelen, Nicholas Walker, Sanghoon Lee, Andrew S. Rosen, Gerbrand Ceder, Kristin Persson, Anubhav Jain  
**arXiv:** [2212.05238](https://arxiv.org/abs/2212.05238)  
**Date:** 2022-12-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2212.05238v1/

## Key points

- Previous work by Huang & Cole fine-tuned a BERT model on battery publications to enhance a database of NLP-extracted battery data.
- Their approach used a question and answer (Q/A) method for limited device-level information extraction.
- Limitations of their approach: couldn't handle passages with multiple devices, required training the BERT language model on a large corpus of battery research papers.
- This work proposes a simple and flexible approach for complex information extraction in scientific text.
- The method involves fine-tuning the GPT-3 language model to perform document-level named entity recognition and relation extraction simultaneously.
- The GPT-3 model generates precisely formatted summaries or structured schemas from text passages like research paper abstracts.
- Researchers only need to define the desired output structure and annotate around 100-500 text passages to use this method.
- The resulting fine-tuned model accurately extracts desired information from text in the same structured representation.
- This approach demonstrates strong performance in both sentence-level and document-level materials information extraction tasks.
- It requires minimal knowledge of how language models work internally, making it accessible even for researchers with little experience in NLP.
- Intermediate models can be used to pre-suggest entities for annotation, speeding up the annotation process for constructing large training sets.
- This method's generality suggests its applicability to other domains such as physics or biology.
- Unlike previous methods relying on fine-tuning on large domain-specific corpora, this approach leverages comprehensive pretraining of language models along with user-provided annotations.

## Summary

In the domain of materials science, Huang & Cole previously fine-tuned a BERT model on battery publications to enhance a database of NLP-extracted battery data. Their approach used a question and answer (Q/A) method to extract limited device-level information. However, their approach had limitations as it couldn't be applied to passages containing information about multiple devices and required training the BERT language model on a large corpus of battery research papers. To address these limitations, this work proposes a simple and flexible approach for complex information extraction in scientific text. The method involves fine-tuning a large language model (GPT-3) to simultaneously perform document-level named entity recognition and relation extraction. This approach can handle complex inter-relations, including hierarchical or list-based information, without the need for enumerating all possible relations or preliminary named entity recognition. The GPT-3 model is trained to accept a text passage, such as a research paper abstract, and generate a precisely formatted summary of the knowledge contained in the prompt. The output can be in the form of English sentences or structured schemas like JSON documents. To use this method, researchers only need to define the desired output structure and annotate around 100-500 text passages accordingly. The resulting fine-tuned model can accurately extract desired information from text and provide output in the same structured representation. This approach demonstrates strong performance in both sentence-level and document-level materials information extraction tasks. Importantly, it requires minimal knowledge of how language models work internally, making it accessible even for researchers with little experience in natural language processing (NLP). Additionally, intermediate models can be used to pre-suggest entities for annotation, significantly speeding up the annotation process for constructing large training sets. While the example tasks presented focus on materials science, this method's generality suggests its applicability to other domains such as physics or biology. Notably, unlike previous methods that relied on fine-tuning on large domain-specific corpora, this approach leverages the comprehensive pretraining of language models along with user-provided annotations to accomplish a wide range of complex tasks. The authors also provide details on their contributions to the development and evaluation of the information extraction method.