# Causal Reasoning through Two Layers of Cognition for Improving Generalization in Visual Question Answering

**Authors:** Trang Nguyen, Naoaki Okazaki  
**arXiv:** [2310.05410](https://arxiv.org/abs/2310.05410)  
**Date:** 2023-10-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.05410v1/

## Key points

- Generalization beyond training distribution is crucial in Visual Question Answering (VQA)
- Previous efforts have focused on refining unimodal aspects, lacking emphasis on enhancing multimodal aspects
- Causal reasoning between interpreting and answering steps in VQA is significant
- Introduction of Cognitive Pathways VQA (CopVQA) to enhance multimodal predictions through causal reasoning factors
- CopVQA operates a pool of pathways capturing diverse causal reasoning flows mirroring human cognition
- Model decomposes responsibility into distinct experts and a cognition-enabled component (CC)
- Two CCs strategically execute one expert for each stage at a time, prioritizing answer predictions governed by pathways involving both CCs
- CopVQA consistently demonstrates improvements in VQA performance and generalization across baselines and domains
- Achieves new state-of-the-art (SOTA) on the PathVQA dataset with comparable accuracy to current SOTA models while utilizing only one-fourth of the model size
- Outperforms other approaches such as SCR in VQA-CPv2 and VQAv2 significantly, achieving comparable or higher accuracy than Mutant without data augmentation
- Qualitative results showcase performance on various datasets like VQA-CPv2 and VQA-RAD, denoted as CopVQA(D) based on DVQA and CFVQA for VQACPv2, as well as CopVQAM based on MMQ for VQARAD
- Focus on causal reasoning through two layers of cognition makes CopVQA a promising advancement in improving generalization in Visual Question Answering tasks

## Summary

, , , , 

In the realm of Visual Question Answering (VQA), the ability to generalize beyond the training distribution is crucial for models to effectively answer questions about images in various contexts. Previous efforts have focused on refining unimodal aspects, but there has been a lack of emphasis on enhancing multimodal aspects. The diverse interpretations of input data lead to different modes of answer generation, underscoring the significance of causal reasoning between interpreting and answering steps in VQA. Introducing Cognitive Pathways VQA (CopVQA), a novel approach that aims to enhance multimodal predictions by emphasizing causal reasoning factors. CopVQA operates a pool of pathways that capture diverse causal reasoning flows throughout interpreting and answering stages, mirroring human cognition. The model decomposes the responsibility of each stage into distinct experts and a cognition-enabled component (CC). Two CCs strategically execute one expert for each stage at a time, prioritizing answer predictions governed by pathways involving both CCs while disregarding answers produced by either CC. Through experiments conducted on real-life and medical data, CopVQA consistently demonstrates improvements in VQA performance and generalization across baselines and domains. Notably, CopVQA achieves a new state-of-the-art (SOTA) on the PathVQA dataset and shows comparable accuracy to current SOTA models on VQA-CPv2, VQAv2, and VQA RAD while utilizing only one-fourth of the model size. Furthermore, when compared to other approaches such as SCR in VQA-CPv2 and VQAv2, CopVQA outperforms these models significantly. Specifically, CopVQA achieves comparable or even higher accuracy than Mutant without data augmentation. Qualitative results also showcase the underlying performance of CopVQA on various datasets like VQA-CPv2 and VQA-RAD, denoted as CopVQA(D) based on DVQA and CFVQA for VQACPv2, as well as CopVQAM based on MMQ for VQARAD. In conclusion, with its focus on causal reasoning through two layers of cognition, CopVQA presents a promising advancement in improving generalization in Visual Question Answering tasks.