# Are Deep Neural Networks SMARTer than Second Graders?

**Authors:** Anoop Cherian, Kuan-Chuan Peng, Suhas Lohit, Kevin Smith, Joshua B. Tenenbaum  
**arXiv:** [2212.09993](https://arxiv.org/abs/2212.09993)  
**Date:** 2022-12-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2212.09993v1/

## Key points

- Deep learning has made remarkable progress in recent times
- Neural networks are being used to solve complex tasks such as playing Go, generating art, and question answering
- The generalizability of these networks when it comes to solving problems that require broad skills is questioned
- A team of researchers led by Anoop Cherian proposed the Simple Multimodal Algorithmic Reasoning Task (SMART) and its associated SMART-101 dataset to address this question
- The dataset comprises 101 unique puzzles designed specifically for children aged 6-8 years old
- Each puzzle requires a mix of several elementary skills such as arithmetic, algebra, and spatial reasoning to solve
- To scale the dataset towards training deep neural networks, the team programmatically generated entirely new instances for each puzzle while retaining their solution algorithm
- Powerful deep models offer reasonable performances on puzzles they are trained on but are not better than random accuracy when analyzed for generalization
- The recent ChatGPT large language model was evaluated on a subset of the dataset and found to produce convincing reasoning abilities but often provided incorrect answers
- The study's authors include Kuan-Chuan Peng, Suhas Lohit, Kevin Smith, Joshua B. Tenenbaum in addition to Anoop Cherian
- Current deep learning models may not be as generalizable as previously thought when it comes to solving problems requiring broad skills like those tested in SMART-101

## Summary

The field of deep learning has seen remarkable progress in recent times, with neural networks being applied to solve complex tasks such as playing Go, generating art, and question answering. However, the question arises as to how generalizable these networks are when it comes to solving problems that require broad skills. To address this question, a team of researchers led by Anoop Cherian proposed the Simple Multimodal Algorithmic Reasoning Task (SMART) and its associated SMART-101 dataset. The dataset comprises 101 unique puzzles designed specifically for children aged 6-8 years old. Each puzzle consists of a picture and a question that requires a mix of several elementary skills such as arithmetic, algebra, and spatial reasoning to solve. To scale the dataset towards training deep neural networks, the team programmatically generated entirely new instances for each puzzle while retaining their solution algorithm. To benchmark the performance on the SMART-101 dataset, the team proposed a vision and language meta-learning model using varied state-of-the-art backbone neural networks. The experiments revealed that while powerful deep models offer reasonable performances on puzzles they are trained on, they are not better than random accuracy when analyzed for generalization. Furthermore, the recent ChatGPT large language model was evaluated on a subset of the dataset and found to produce convincing reasoning abilities but often provided incorrect answers. The study's authors include Kuan-Chuan Peng, Suhas Lohit, Kevin Smith, and Joshua B. Tenenbaum in addition to Anoop Cherian. Their findings suggest that current deep learning models may not be as generalizable as previously thought when it comes to solving problems requiring broad skills like those tested in SMART-101.