# Beyond English: Evaluating LLMs for Arabic Grammatical Error Correction

**Authors:** Sang Yun Kwon, Gagan Bhatia, El Moatez Billah Nagoudi, Muhammad Abdul-Mageed  
**arXiv:** [2312.08400](https://arxiv.org/abs/2312.08400)  
**Date:** 2023-12-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.08400v1/

## Key points

- Study evaluates performance of instruction finetuned large language models (LLMs) in Arabic grammatical error correction (GEC)
- LLMs have shown promising results in English natural language processing tasks, but effectiveness for other languages not extensively explored
- Arabic's rich morphology makes GEC challenging
- Researchers employ different prompting methods and utilize few-shot learning to enhance model's performance
- GPT-4 achieves F1 score of up to 65.49 under expert prompting, 5 points higher than established baseline
- Smaller fully finetuned models outperform instruction finetuned ones, indicating room for improvement in LLMs
- Researchers develop method using synthetic data inspired by low-resource machine translation techniques
- Method surpasses previous models' performance on two standard Arabic benchmarks
- Best model achieves new state-of-the-art result with F1 scores of 73.29 and 73.26 on 2014 and 2015 QALB datasets respectively compared to peer-reviewed published baselines
- Research highlights potential and limitations of instruction finetuned LLMs in Arabic GEC
- Introduces novel approach that improves upon existing methods for this task

## Summary

This study evaluates the performance of instruction finetuned large language models (LLMs) in Arabic grammatical error correction (GEC). While LLMs have shown promising results in English natural language processing tasks, their effectiveness for other languages has not been extensively explored. Arabic's rich morphology makes it a challenging task. The researchers employ different prompting methods and utilize few-shot learning to enhance the model's performance. They find that GPT-4 achieves an F1 score of up to 65.49 under expert prompting, which is approximately 5 points higher than the established baseline. However, they observe that even smaller fully finetuned models outperform instruction finetuned ones, indicating room for improvement in LLMs. Inspired by low-resource machine translation techniques, the researchers develop a method using synthetic data which surpasses previous models' performance on two standard Arabic benchmarks. Their best model achieves a new state-of-the-art result with F1 scores of 73.29 and 73.26 on the 2014 and 2015 QALB datasets respectively compared to peer-reviewed published baselines. This research highlights both the potential and limitations of instruction finetuned LLMs in Arabic GEC and introduces a novel approach that improves upon existing methods for this task.