# Measuring Massive Multitask Language Understanding

**Authors:** Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt  
**arXiv:** [2009.03300](https://arxiv.org/abs/2009.03300)  
**Date:** 2020-09-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2009.03300v1/

## Key points

- Authors propose a new test to evaluate multitask accuracy of text models
- Test encompasses 57 tasks spanning various domains
- Achieving high accuracy on the test requires extensive world knowledge and problem-solving abilities
- Most recent models exhibit near random-chance accuracy on the test
- Largest GPT-3 model shows improvement of nearly 20 percentage points above random chance on average
- Best models still require significant enhancements to achieve human-level accuracy across all tasks
- Models often display lopsided performance and struggle to recognize when they are incorrect
- Text models still exhibit near-random accuracy in socially important subjects like morality and law
- Proposed test provides a valuable framework to assess academic and professional understanding across multiple tasks and identify critical shortcomings effectively
- Study highlights limitations of current text models in terms of multitask accuracy
- Further advancements needed to bridge the gap between machine performance and human-level comprehension.

## Summary

In their paper titled "Measuring Massive Multitask Language Understanding," authors Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt propose a new test to evaluate the multitask accuracy of text models. This test encompasses 57 tasks spanning various domains such as elementary mathematics, US history, computer science, law and more. The authors emphasize that achieving high accuracy on this test requires models to possess extensive world knowledge and problem-solving abilities. The study reveals that while most recent models exhibit near random-chance accuracy on the test, the largest GPT-3 model demonstrates an improvement of nearly 20 percentage points above random chance on average. However, despite this progress, the best models still require significant enhancements before they can achieve human-level accuracy across all 57 tasks. Additionally, these models often display lopsided performance and struggle to recognize when they are incorrect. Furthermore, the research highlights that even in socially important subjects like morality and law; text models still exhibit near-random accuracy. To comprehensively assess a model's academic and professional understanding across multiple tasks and identify critical shortcomings effectively; the proposed test provides a valuable framework. Overall; this study sheds light on the limitations of current text models in terms of multitask accuracy and underscores the need for further advancements to bridge the gap between machine performance and human-level comprehension.