# Low-Resource Language Modelling of South African Languages

**Authors:** Stuart Mesham, Luc Hayward, Jared Shapiro, Jan Buys  
**arXiv:** [2104.00772](https://arxiv.org/abs/2104.00772)  
**Date:** 2021-04-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2104.00772v1/

## Key points

- Researchers address challenges in developing language models for African languages due to limited research and resources
- Lack of standardized training and evaluation sets for high-resource languages like English complicates the task
- Evaluation of open-vocabulary language models on low-resource South African languages using byte-pair encoding to handle rich morphology
- Experimentation with various models including n-gram models, feedforward neural networks, RNNs, and Transformers on small-scale datasets from isiZulu and Sepedi languages
- Well-regularized RNNs outperform other models on two isiZulu datasets and one Sepedi dataset
- Multilingual training significantly improves performance on these datasets
- Use of byte pair encoding effective in controlling vocabulary size and enabling open-vocabulary language modeling
- Research opens up new possibilities for exploring multilingual and low-resource language modeling for African languages

## Summary

In their study "Low-Resource Language Modelling of South African Languages," researchers Stuart Mesham, Luc Hayward, Jared Shapiro, and Jan Buys address the challenges faced in developing language models for African languages due to limited research and resources. The lack of standardized training and evaluation sets for high-resource languages like English further complicates the task. To overcome this issue, the researchers evaluate the performance of open-vocabulary language models on low-resource South African languages using byte-pair encoding to handle their rich morphology. They experiment with various models including n-gram models, feedforward neural networks, recurrent neural networks (RNNs), and Transformers on small-scale datasets from isiZulu and Sepedi languages. The results demonstrate that well-regularized RNNs outperform other models on two isiZulu datasets and one Sepedi dataset. Furthermore, multilingual training significantly improves performance on these datasets. The use of byte pair encoding proves effective in controlling vocabulary size and enabling open-vocabulary language modeling. This research opens up new possibilities for exploring multilingual and low-resource language modeling for African languages. By shedding light on the performance of different language models on South African languages, this study paves the way for future advancements in natural language understanding and generation in underrepresented linguistic communities.