# Foundation Models for Remote Sensing and Earth Observation: A Survey

**Authors:** Aoran Xiao, Weihao Xuan, Junjue Wang, Jiaxing Huang, Dacheng Tao, Shijian Lu, Naoto Yokoya  
**arXiv:** [2410.16602](https://arxiv.org/abs/2410.16602)  
**Date:** 2024-10-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.16602v1/

## Key points

- Remote Sensing (RS) is crucial for observing and interpreting the planet, with applications in various fields.
- Artificial intelligence (AI), particularly deep learning, has made strides in RS but faces challenges due to Earth's complexity and diverse sensor modalities.
- Recent advancements in large Foundation Models (FMs) have led to the emergence of Remote Sensing Foundation Models (RSFMs) tailored for Earth Observation tasks.
- Challenges in developing RSFMs include domain discrepancies, limited pre-training datasets, lack of specialized architectures, and unique RS applications.
- Efforts are underway to address these challenges by developing advanced RSFMs and integrating FMs within the RS domain.
- The paper provides a comprehensive survey of recent advancements in RSFMs categorized into Visual Foundation Models (VFMs), Visual-Language Models (VLMs), Large Language Models (LLMs), and generative FMs for RS.
- Key contributions include a systematic review of advancements in RSFMs across different model types and sensor modalities, benchmarking performance on various tasks, and identifying research challenges for future exploration.

## Summary

, , , , 

Remote Sensing (RS) plays a crucial role in observing, monitoring, and interpreting our planet, with applications spanning geoscience, economics, humanitarian fields, and more. Artificial intelligence (AI), particularly deep learning, has made significant strides in RS but faces challenges due to the complexity of Earth's environments, diverse sensor modalities, and varying resolutions. Recent advancements in large Foundation Models (FMs) have shown promise in various domains but struggle with RS data of non-optical modalities. This has led to the emergence of Remote Sensing Foundation Models (RSFMs) tailored for Earth Observation (EO) tasks. Developing RSFMs presents challenges such as domain discrepancies between natural and RS data, limited pre-training datasets, lack of specialized architectures, and unique RS applications. Efforts are underway to address these challenges by developing advanced RSFMs and integrating FMs within the RS domain. However, the field lacks a comprehensive survey on RSFMs. This paper aims to fill this gap by providing an extensive survey of recent advancements in RSFMs. It categorizes existing methods into Visual Foundation Models (VFMs), Visual-Language Models (VLMs), Large Language Models (LLMs), and generative FMs for RS. The survey covers learning paradigms, datasets, technical approaches, benchmarks, and future research directions. Key contributions include a systematic review of recent advancements in RSFMs across different model types and sensor modalities. The paper benchmarks and analyzes the performance of RSFMs on various tasks and identifies research challenges for future exploration. The structure of the survey includes background knowledge on RSFMs in Section 2, foundations of RSFMs in Section 3, reviews of VFMs in Section 4, VLMs in Section 5, other types of RSFMs in Section 6. Performance comparisons across benchmark datasets are presented in Section 7 with future research directions outlined in Section 8. Additionally, early insights from Manvi et al. revealed that LLMs possess spatial knowledge but struggle with accurate predictions for geospatial indicators like population density. To address this limitation, GeoLLM was introduced to fine-tune LLMs using prompts enriched with auxiliary map data from OpenStreetMap. Generative models for RS have also been explored for image generation tasks like inpainting and colorization but face challenges due to the unique characteristics of multi-spectral RS data. Overall, this detailed summary highlights the importance of developing specialized AI models like RSFMs for effectively utilizing large-scale geospatial data and addressing complex Earth surface dynamics while offering insights into current advancements and future research directions in this rapidly evolving field.