# A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity

**Authors:** Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, Pascale Fung  
**arXiv:** [2302.04023](https://arxiv.org/abs/2302.04023)  
**Date:** 2023-02-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.04023v4/

## Key points

- Comprehensive framework for evaluating Interactive Large Language Models (LLMs) like ChatGPT
- Performance assessment across 23 publicly available datasets covering 8 common NLP tasks
- Introduction of a new multimodal dataset for evaluation
- ChatGPT outperforms other LLMs in zero-shot learning and some fine-tuned models
- Struggles with generating non-Latin script languages and faces hallucination issues due to limited access to external knowledge bases
- Evaluation of reasoning abilities, proficiency in logical, non-textual, and commonsense reasoning tasks
- Unreliable reasoner, better in deductive than inductive reasoning tasks
- Human collaboration through prompt engineering strategies to enhance performance
- Details on various NLP tasks such as summarization, machine translation (MT), sentiment analysis, question answering, misinformation detection, task-oriented dialogue systems, and open-domain dialogue systems
- Emphasis on ethical considerations in generative AI model development: fairness, toxicity, demographic bias, safety
- Acknowledgment of funding sources including government grants in Hong Kong
- Compliance with data usage licenses and terms outlined by OpenAI and other relevant entities involved

## Summary

This paper presents a comprehensive framework for quantitatively evaluating Interactive Large Language Models (LLMs) such as ChatGPT. The authors assess the model's performance across 23 publicly available datasets covering 8 common Natural Language Processing (NLP) tasks. They also introduce a new multimodal dataset to evaluate the model's capabilities in this area. Results show that ChatGPT outperforms other LLMs in zero-shot learning and even surpasses fine-tuned models in some cases. However, it struggles with generating non-Latin script languages and faces hallucination issues due to limited access to external knowledge bases. The study also evaluates ChatGPT's reasoning abilities and shows its proficiency in logical, non-textual, and commonsense reasoning tasks. However, it is noted to be an unreliable reasoner and performs better in deductive than inductive reasoning tasks. The interactive feature of ChatGPT allows for human collaboration through prompt engineering strategies to enhance its performance. The paper provides details on various NLP tasks such as summarization, machine translation (MT), sentiment analysis, question answering, misinformation detection, task-oriented dialogue systems, and open-domain dialogue systems. It also includes evaluation metrics for comparison with state-of-the-art (SOTA), fine-tuned models, and zero-shot learning approaches. In light of ethical considerations surrounding generative AI models like ChatGPT, responsible design and usage are highlighted as crucial challenges that require ongoing research efforts. The authors emphasize the importance of addressing issues related to fairness, toxicity, demographic bias, and safety in LLM development. Finally,from various sources including government grants in Hong Kong is acknowledged for this research endeavor. Compliance with data usage licenses and terms outlined by OpenAI and other relevant entities involved in the study is also underscored.