# Evaluating ChatGPT-4 Vision on Brazil's National Undergraduate Computer Science Exam

**Authors:** Nabor C. Mendonça  
**arXiv:** [2406.09671](https://arxiv.org/abs/2406.09671)  
**Date:** 2024-06-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.09671v1/

## Key points

- Study evaluated ChatGPT-4 Vision performance on Brazil's 2021 National Undergraduate Exam (ENADE) for Computer Science
- ChatGPT-4 Vision outperformed average exam participant, ranking within top 10 percentile
- Challenges faced in question interpretation, logical reasoning, and visual acuity
- Utilized generative AI tools like ChatGPT-4, GitHub Copilot, and Gemini Advanced for manuscript revision and data analysis code generation
- Evaluation of Visual Language Models (VLMs) highlighted challenges in visual deductive reasoning tasks
- Insights from previous studies can enhance understanding of ChatGPT-4 Vision's capabilities in multimodal reasoning tasks beyond visuals
- Suggested developing MathVerse-like benchmark based on ENADE questions to further evaluate model's performance in real-world educational assessments
- Integration of visual capabilities into Language Learning Models (LLMs) revolutionizing natural language processing and enhancing education in science and technology fields
- Human oversight crucial to verify accuracy of models like ChatGPT-4 Vision and ensure fairness in high-stakes exams

## Summary

In this study, we evaluated the performance of ChatGPT-4 Vision on Brazil's 2021 National Undergraduate Exam (ENADE) for Computer Science. ChatGPT-4 Vision is an advanced visual model developed by OpenAI. By presenting the model with exam questions in their original image format and allowing for reassessment with different answer keys, we assessed its reasoning and self-reflective capabilities in a large-scale academic assessment involving textual and visual content. Our findings showed that ChatGPT-4 Vision outperformed the average exam participant, ranking within the top 10 percentile. However, it faced challenges in question interpretation, logical reasoning, and visual acuity. This study contributes to existing research by exploring ChatGPT-4 Vision's self-reflective abilities on high-stakes academic exams. We utilized generative AI tools like ChatGPT-4, GitHub Copilot, and Gemini Advanced for manuscript revision, table creation, figure formatting, and data analysis code generation. The evaluation of Visual Language Models (VLMs) by Zhang et al. highlighted challenges in visual deductive reasoning tasks and emphasized the need for improved spatial reasoning techniques proposed by Mayne and Wu. While our work differs from previous studies focusing on visual deductive reasoning tasks, insights from these studies can enhance our understanding of ChatGPT-4 Vision's capabilities in multimodal reasoning tasks beyond visuals. To further evaluate the model's performance in real-world educational assessments, we suggest developing a MathVerse-like benchmark based on ENADE questions. The integration of visual capabilities into Language Learning Models (LLMs) has revolutionized natural language processing and holds promise for enhancing education in science and technology fields. However, human oversight remains crucial to verify the accuracy of models like ChatGPT-4 Vision and ensure fairness in high-stakes exams. Our findings underscore the importance of improved question design to enhance model performance and highlight areas where further research is needed to maximize the potential of multimodal AI models in educational assessments.