# Benchmarking Large Language Models for Persian: A Preliminary Study Focusing on ChatGPT

**Authors:** Amirhossein Abaskohi, Sara Baruni, Mostafa Masoudi, Nesa Abbasi, Mohammad Hadi Babalou, Ali Edalat, Sepehr Kamahi, Samin Mahdizadeh Sani, Nikoo Naghavian, Danial Namazifard, Pouya Sadeghi, Yadollah Yaghoobzadeh  
**arXiv:** [2404.02403](https://arxiv.org/abs/2404.02403)  
**Date:** 2024-04-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.02403v1/

## Key points

- Large language models (LLMs) like ChatGPT and GPT-3.5-turbo have shown impressive performance in English but their effectiveness in low-resource languages like Persian is uncertain.
- A comprehensive benchmarking analysis of LLMs, including GPT-3.5-turbo, GPT-4, and OpenChat-3.5, was conducted across various tasks in Persian.
- The study introduced new benchmarks for reasoning tasks in Persian based on elementary school math questions and entrance exams for 7th and 10th grades due to the scarcity of Persian datasets.
- LLMs often fall short compared to smaller pre-trained models fine-tuned for specific tasks in Persian.
- Improved performance was observed when test sets were translated to English before being inputted into GPT-3.5, suggesting potential enhancements in LLM performance for the Persian language context.
- Leveraging LLMs shows promise for enhancing natural language processing capabilities in Persian based on the study's findings.

## Summary

This paper delves into the effectiveness of large language models (LLMs) for the Persian language, with a specific focus on ChatGPT and other subsequent LLMs. While these models have showcased impressive performance in English, their applicability to low-resource languages like Persian remains uncertain. To address this gap, the study conducts a comprehensive benchmarking analysis of LLMs across various tasks in Persian. The primary model under scrutiny is GPT-3.5-turbo, although evaluations also encompass GPT-4 and OpenChat-3.5 to provide a more holistic assessment. The research covers a wide range of tasks categorized into classic, reasoning, and knowledge-based domains. In order to facilitate a thorough comparison, LLMs are pitted against task-specific fine-tuned models that already exist. Given the scarcity of Persian datasets for reasoning tasks, the study introduces two new benchmarks: one based on elementary school math questions and another derived from entrance exams for 7th and 10th grades. The findings indicate that while LLMs , they often fall short compared to smaller pre-trained models fine-tuned for specific tasks. Furthermore, an interesting observation is made regarding improved performance when test sets are translated to English before being inputted into GPT-3.5. This suggests potential enhancements in LLM performance for the Persian language context. Notably, this holds significance due to . In light of these results, it becomes evident that there is significant promise in leveraging LLMs for enhancing natural language processing capabilities in Persian. This study contributes valuable insights towards understanding .