Recent growth in remote sensing collection capacity and spectral diversity has produced vast volumes of data, yet traditional AI/ML algorithms struggle to generalize across modalities, targets, or geographies. These approaches are typically fine-tuned for specific applications over months and operate on a single modality, limiting adaptability to real-time mission requirements where certain sensor data may be unavailable. Traditional multimodal data fusion methods improve performance by combining outputs from independent models, but they still require co-collected imagery for training and inference, which is often impractical for time-critical applications. We propose Cross-Modal Foundation Models to address these limitations. These models are pretrained on all available data using contrastive learning, without requiring manual annotations, to generate consistent feature embeddings across modalities. By learning modality-invariant “fingerprints”, the model can operate even when only partial data is available, enabling inference with a wide range of sensors regardless of perspective, resolution, or modality. This adaptability allows the model to leverage the strengths of different sensors: SAR data in cloudy conditions, EO imagery when color information is critical, or a combination of available modalities for optimal performance. It also relieves constraints on sensor tasking, enabling sensors to be dynamically allocated across missions without compromising intelligence value. In this study, we present a cross-modal foundation model aligning feature spaces between SAR and EO imagery, providing a versatile framework for downstream tasks such as change detection, activity recognition, and automatic target recognition. Beyond immediate applications, this framework lays the groundwork for future automated sensor orchestration, allowing systems to identify missing information and optimize data collection for maximum situational awareness and operational efficiency.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Cross-Modal Foundation Models for Remote Sensing

  • Adam Francisco,
  • Matthew D. Reisman,
  • Tobe Corazzini,
  • Ryan McCormick,
  • Kevin J. LaTourette

摘要

Recent growth in remote sensing collection capacity and spectral diversity has produced vast volumes of data, yet traditional AI/ML algorithms struggle to generalize across modalities, targets, or geographies. These approaches are typically fine-tuned for specific applications over months and operate on a single modality, limiting adaptability to real-time mission requirements where certain sensor data may be unavailable. Traditional multimodal data fusion methods improve performance by combining outputs from independent models, but they still require co-collected imagery for training and inference, which is often impractical for time-critical applications. We propose Cross-Modal Foundation Models to address these limitations. These models are pretrained on all available data using contrastive learning, without requiring manual annotations, to generate consistent feature embeddings across modalities. By learning modality-invariant “fingerprints”, the model can operate even when only partial data is available, enabling inference with a wide range of sensors regardless of perspective, resolution, or modality. This adaptability allows the model to leverage the strengths of different sensors: SAR data in cloudy conditions, EO imagery when color information is critical, or a combination of available modalities for optimal performance. It also relieves constraints on sensor tasking, enabling sensors to be dynamically allocated across missions without compromising intelligence value. In this study, we present a cross-modal foundation model aligning feature spaces between SAR and EO imagery, providing a versatile framework for downstream tasks such as change detection, activity recognition, and automatic target recognition. Beyond immediate applications, this framework lays the groundwork for future automated sensor orchestration, allowing systems to identify missing information and optimize data collection for maximum situational awareness and operational efficiency.