# Foundational Models Defining a New Era in Vision: A Survey and Outlook

**Authors:** Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Fahad Shahbaz Khan  
**arXiv:** [2307.13721](https://arxiv.org/abs/2307.13721)  
**Date:** 2023-07-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.13721v1/

## Key points

- Significant progress in developing foundational models for understanding and reasoning about visual scenes
- Bridging the gap between different modalities (vision, text, audio, depth) for contextual reasoning and generalization
- Modification of models through human-provided prompts without retraining
- Comprehensive review of emerging foundational models including architecture designs, training objectives, pre-training datasets, fine-tuning mechanisms, and prompting patterns
- Open challenges and research directions including evaluation difficulties, real-world understanding gaps, contextual limitations, biases vulnerability to adversarial attacks, and interpretability issues
- Wide range of applications of foundation models and a list of studied models provided for reference
- Recent developments in scaling up foundation models using large language models (LLMs) with billions of parameters
- Effectiveness of scaled models in zero/few-shot learning and achieving state-of-the-art performance on challenging problems
- Detailed overview of foundational models in computer vision presented in the survey
- Outline of future directions for research and development in this area.

## Summary

In recent years, there has been significant progress in developing foundational models that can understand and reason about visual scenes. These models bridge the gap between different modalities such as vision, text, audio, and depth to facilitate contextual reasoning and generalization. They can also be modified through human-provided prompts without retraining, allowing for tasks like object segmentation or interactive dialogues. In this survey, the authors provide a comprehensive review of emerging foundational models including their architecture designs, training objectives, pre-training datasets, fine-tuning mechanisms and prompting patterns. They also discuss the open challenges and research directions in this field such as evaluation difficulties, real-world understanding gaps, contextual limitations, biases vulnerability to adversarial attacks and interpretability issues. The survey covers a wide range of applications of foundation models and provides a list of studied models for reference. Additionally, the authors highlight recent developments in scaling up foundation models using large language models (LLMs) with billions of parameters. They demonstrate the effectiveness of these scaled models in zero/few-shot learning and achieving state-of-the-art performance on various challenging problems. Overall this survey presents a detailed overview of foundational models in computer vision and outlines future directions for research and development in this area.