# Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settings

**Authors:** Krishno Dey, Prerona Tarannum, Md. Arid Hasan, Imran Razzak, Usman Naseem  
**arXiv:** [2410.13153](https://arxiv.org/abs/2410.13153)  
**Date:** 2024-10-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.13153v1/

## Key points

- Large Language Models (LLMs) have shown remarkable performance in natural language processing tasks
- LLMs are predominantly developed and evaluated in English, leading to a gap in understanding their effectiveness in low-resource languages like Bangla, Hindi, and Urdu
- This study evaluates the performance of LLMs such as GPT-4, Llama 2, and Gemini across English and low-resource languages
- Traditional machine learning models and transformer-based approaches have been used for analyzing low-resource languages, but multi-lingual LLMs offer new opportunities
- Computational resources for Bangla, Hindi, and Urdu are limited despite being widely spoken globally
- The study focuses on evaluating the effectiveness of LLMs specifically in Bangla, Hindi, and Urdu compared to English
- Promising results with LLMs in these languages have been observed but more comprehensive studies are needed to determine their full potential
- Zero-shot prompting and different prompt settings are utilized to analyze how GPT-4 outperforms other LLMs across all five prompt settings and languages
- While all three models perform better with English prompts, there is room for improvement with low-resource language prompts
- The study contributes to enhancing LLM capabilities in addressing challenges posed by low-resource languages like Bangla, Hindi, and Urdu

## Summary

In recent years, Large Language Models (LLMs) have gained significant attention for their remarkable performance in various natural language processing tasks. However, most LLMs are predominantly developed and evaluated in English, leaving a gap in understanding their effectiveness in low-resource languages, particularly those spoken widely in South Asia such as Bangla, Hindi, and Urdu. To address this gap, this study evaluates the performance of LLMs like GPT-4, Llama 2, and Gemini across English and these low-resource languages. Previous research has laid the groundwork for exploring LLMs in downstream tasks for low-resource languages. While traditional machine learning models and transformer-based approaches have been commonly used for analyzing these languages, the emergence of multi-lingual LLMs presents new opportunities. Despite being some of the most spoken languages globally, computational resources for Bangla, Hindi, and Urdu remain limited. The study focuses on evaluating the effectiveness of LLMs specifically in Bangla, Hindi, and Urdu compared to English. Existing literature showcases promising results with LLMs in these languages but highlights the need for more comprehensive studies to determine their full potential and identify areas for improvement. By utilizing zero-shot prompting and different prompt settings, the study aims to provide a detailed analysis of how GPT-4 outperforms other LLMs like Llama 2 and Gemini across all five prompt settings and languages. The findings suggest that while all three models perform better with English prompts, there is room for enhancing their performance with low-resource language prompts. Overall, this study contributes to the ongoing efforts to enhance LLMs' capabilities in addressing the unique challenges posed by low-resource languages like Bangla, Hindi, and Urdu. By shedding light on the strengths and limitations of current models in cross-lingual contexts, it paves the way for future advancements in developing more inclusive NLP applications tailored to diverse linguistic landscapes.