MRF: A Modality-Resilient Framework for Handling Missing Modalities in Multimodal Learning
摘要
Multimodal learning has emerged as a pivotal advancement in artificial intelligence by integrating textual and non-textual data to enhance both understanding and generative capabilities. However, the challenge of missing modalities during training often leads to degraded performance, thereby limiting the applicability of these models in real-world scenarios. In this paper, we propose a comprehensive solution to the problem of missing modalities in multimodal learning. Our approach introduces a Modality-Resilient Framework (MRF) that integrates modality-invariant feature learning, cross-modal attention mechanisms, and an advanced imputation strategy that leverages inter-modal relationships. Extensive experiments on benchmark datasets demonstrate that the proposed framework not only outperforms existing methods but also substantially enhances robustness in the presence of incomplete modality inputs.