# Unifying (Machine) Vision via Counterfactual World Modeling

**Authors:** Daniel M. Bear, Kevin Feigelis, Honglin Chen, Wanhee Lee, Rahul Venkatesh, Klemen Kotar, Alex Durango, Daniel L. K. Yamins  
**arXiv:** [2306.01828](https://arxiv.org/abs/2306.01828)  
**Date:** 2023-06-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.01828v1/

## Key points

- Different approaches in machine vision often rely on costly labeled datasets and lack robustness
- "Foundation models" in natural language processing have shown effectiveness without task-specific training
- Counterfactual World Modeling (CWM) is a framework for constructing a visual foundation model
- CWM consists of two key components: structured masking and counterfactual prompting
- Structured masking encourages the model to capture low-dimensional structure in visual data
- Counterfactual prompting allows the model to compute distinct visual representations by comparing outputs on real and modified inputs
- CWM produces high-quality results for tasks such as estimating keypoints, optical flow, occlusions, object segments, and relative depth
- CWM has the potential to unify different strands of machine vision into a conceptually simple foundation
- CWM offers a promising path towards addressing complexity and limitations in current machine vision approaches

## Summary

In the field of machine vision, different approaches have been used to tackle specific tasks, but these methods often rely on costly labeled datasets and lack robustness in general perception. However, in natural language processing, "foundation models" have demonstrated the effectiveness of large pre-trained neural networks in solving a wide range of tasks without task-specific training. To bridge this gap between language and vision, the authors propose Counterfactual World Modeling (CWM), a framework for constructing a visual foundation model. CWM consists of two key components that address the challenges faced by previous attempts to apply foundation models to vision. The first component is structured masking, which extends masked prediction methods to encourage the prediction model to capture the low-dimensional structure in visual data. This allows the model to identify and represent the essential physical components of a scene through small sets of visual tokens. The second component is counterfactual prompting, which leverages the observation that many distinct visual representations can be computed by comparing the prediction model's output on real inputs with slightly modified ("counterfactual") inputs. This zero-shot approach enables CWM to perform various visual computations without task-specific training. The authors demonstrate that CWM produces high-quality results for diverse tasks such as estimating keypoints, optical flow, occlusions, object segments, and relative depth on real-world images and videos. These findings suggest that CWM has the potential to unify different strands of machine vision into a conceptually simple foundation. Overall, CWM offers a promising path towards addressing the complexity and limitations in current machine vision approaches by providing a unified unsupervised network capable of performing a wide variety of visual computations.