Multimodal data fusion has shown great success in real-world applications by leveraging complementary cues across modalities, enhancing feature robustness, and mitigating individual limitations. In the context of aerial robotics and drone-based remote sensing, modalities such as synthetic aperture radar (SAR), hyperspectral imagery (HSI), optical, and light detection and ranging (LiDAR) provide distinct scene perspectives. However, integration of such modalities effectively remains challenging due to variations in spatial resolution, spectral properties, and noise patterns. Traditional multimodal fusion methods often struggle to capture both modality-specific and cross-modal features, limiting their ability to exploit complementary information fully. The attention-driven multimodal fusion framework incorporating self-attention and cross-attention mechanisms to address this. Self-attention refines modality-specific features by highlighting the most relevant spatial and spectral information, while cross-attention facilitates inter-modal alignment, enabling effective information exchange. Also, two auxiliary tasks that have been incorporated for modality-specific classification, producing highly discriminative cross-attention masks. To further enhance feature independence, the model penalizes high correlations between attended modality-specific features, ensuring non-redundant representations. The approach enhances the latent space by capturing local and global dependencies, improving classification performance. We discuss the experiments performed on five benchmark multimodal RS datasets, including optical, multispectral (MS), HSI, LiDAR, SAR, and audio modalities, including Houston 2013 HSI-MSI, HSI-LiDAR, Berlin HSI-SAR, Augsburg HSI-SAR, and ADVANCE. The results evidence the effectiveness of attention-driven fusion strategy in multimodal learning, contributing to more generalizable and interpretable fusion frameworks for drone-based remote sensing and aerial robotics.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Enhancing Latent Space with Self and Cross-Attention in Multimodal Remote Sensing Image Classification

  • Ankit Jha,
  • Mainak Singha,
  • Girish Mishra,
  • Biplab Banerjee

摘要

Multimodal data fusion has shown great success in real-world applications by leveraging complementary cues across modalities, enhancing feature robustness, and mitigating individual limitations. In the context of aerial robotics and drone-based remote sensing, modalities such as synthetic aperture radar (SAR), hyperspectral imagery (HSI), optical, and light detection and ranging (LiDAR) provide distinct scene perspectives. However, integration of such modalities effectively remains challenging due to variations in spatial resolution, spectral properties, and noise patterns. Traditional multimodal fusion methods often struggle to capture both modality-specific and cross-modal features, limiting their ability to exploit complementary information fully. The attention-driven multimodal fusion framework incorporating self-attention and cross-attention mechanisms to address this. Self-attention refines modality-specific features by highlighting the most relevant spatial and spectral information, while cross-attention facilitates inter-modal alignment, enabling effective information exchange. Also, two auxiliary tasks that have been incorporated for modality-specific classification, producing highly discriminative cross-attention masks. To further enhance feature independence, the model penalizes high correlations between attended modality-specific features, ensuring non-redundant representations. The approach enhances the latent space by capturing local and global dependencies, improving classification performance. We discuss the experiments performed on five benchmark multimodal RS datasets, including optical, multispectral (MS), HSI, LiDAR, SAR, and audio modalities, including Houston 2013 HSI-MSI, HSI-LiDAR, Berlin HSI-SAR, Augsburg HSI-SAR, and ADVANCE. The results evidence the effectiveness of attention-driven fusion strategy in multimodal learning, contributing to more generalizable and interpretable fusion frameworks for drone-based remote sensing and aerial robotics.