# GlossLM: Multilingual Pretraining for Low-Resource Interlinear Glossing

**Authors:** Michael Ginn, Lindia Tjuatja, Taiqi He, Enora Rice, Graham Neubig, Alexis Palmer, Lori Levin  
**arXiv:** [2403.06399](https://arxiv.org/abs/2403.06399)  
**Date:** 2024-03-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.06399v1/

## Key points

- Annotated text in formats like interlinear glossed text (IGT) is crucial for detailed morphosyntactic analyses in a morpheme-by-morpheme format.
- Previous research has focused on automating the generation of IGT to streamline language analysis processes.
- Many languages, especially those needing preservation, lack sufficient IGT data for effective model training.
- Crosslingual transfer has been proposed as a solution to address the lack of IGT data for low-resource languages.
- A comprehensive corpus of over 450k IGT examples across 1.8k languages has been compiled to facilitate research on crosslingual transfer and IGT generation.
- Pretraining a large multilingual model on a portion of the corpus followed by fine-tuning demonstrates competitiveness with state-of-the-art methods for segmented data and large monolingual datasets.
- The model outperforms existing models on unsegmented text and small corpora by up to 6.6% in morpheme accuracy, showcasing the effectiveness of crosslingual transfer for low-resource languages.
- Annotated text aids in preserving minority languages by creating reference materials such as dictionaries and grammars.
- Pretrained models available through platforms like Hugging Face enhance accessibility for researchers and practitioners involved in language documentation efforts.

## Summary

In the field of language documentation, the creation of annotated text in formats like interlinear glossed text (IGT) plays a crucial role in capturing detailed morphosyntactic analyses in a morpheme-by-morpheme format. Previous research has focused on developing methods to automatically generate IGT to streamline the process of language analysis. However, many languages, especially those in need of preservation, lack sufficient IGT data to train effective models. To address this limitation, crosslingual transfer has been proposed as a solution. To facilitate research on crosslingual transfer and IGT generation, a comprehensive corpus of IGT data from various sources has been compiled, encompassing over 450k examples across 1.8k languages. This extensive dataset enables the pretraining of a large multilingual model on a portion of the corpus, followed by further fine-tuning for specific languages. The resulting model demonstrates competitiveness with state-of-the-art methods for segmented data and large monolingual datasets. Moreover, the model outperforms existing models on unsegmented text and small corpora by up to 6.6% in morpheme accuracy, showcasing the effectiveness of crosslingual transfer for low-resource languages. With nearly half of the world's 7,000 languages facing endangerment, efforts to preserve and revitalize minority languages are paramount. Annotated text plays a vital role in these preservation endeavors by aiding in the creation of reference materials such as dictionaries and grammars. The availability of pretrained models through platforms like Hugging Face provides accessibility for researchers and practitioners involved in language documentation efforts. Inspired by previous works on automatic interlinear glossing and large multilingual pretrained models, this study contributes valuable insights into leveraging crosslingual transfer for enhancing language analysis capabilities across diverse linguistic contexts.