Synthesizing Audio-Visual Embeddings for Few-Shot Classification with Limited Modalities in Aerial Sensing
摘要
Aerial sensing (AS) recognition have gained increasing interest due to their wide-ranging applications in environmental monitoring, disaster response, and geospatial analysis. The recent advancements in deep learning have significantly improved recognition performance; however, training neural networks solely on unimodal AS visual data presents several challenges, such as occlusion, intra-class variability, and lighting inconsistencies. While multi-modal learning, particularly the fusion of audio and visual information has been shown to enhance classification performance in low-data regimes, its potential remains largely unexplored in aerial sensing and drone applications. This is particularly relevant given the practical difficulties in acquiring multi-modal data due to sensor limitations, atmospheric conditions, or system failures. To address these challenges, we introduce a novel problem formulation within the few-shot learning (FSL) paradigm, where both audio and visual modalities are available during the meta-training phase, but one modality may be absent during meta-testing. To address this challenge, HAVE-Net (Jha et al. Joint European conference on machine learning and knowledge discovery in databases, Springer, 2023) proposes a generative framework that learns to synthesize missing cross-modal features, enabling robust classification even in the presence of incomplete data. Specifically, our approach employs a modality hallucination strategy that meta-learns to generate plausible missing features from base classes, allowing the model to generalize effectively to novel classes with limited supervision. By leveraging this cross-modal augmentation technique, the authors in Jha et al. (Joint European conference on machine learning and knowledge discovery in databases, Springer, 2023) enhance the model’s ability to perform few-shot classification under real-world constraints. Extensive experiments on the ADVANCE and AudioSetZSL benchmark datasets demonstrate the efficacy of our proposed framework. Our approach consistently outperforms models trained solely on real multimodal data, highlighting the benefits of modality hallucination in improving classification accuracy. These findings underscore the importance of robust cross-modal learning strategies for AS and drone applications, paving the way for more resilient and adaptive recognition models in challenging operational environments.