Medical Vision Language Model With Multi-granularity Alignment and Learning Data Augmentation
摘要
Medical vision-language pretraining (MVLP) has emerged as a powerful paradigm for learning joint representations of medical images and reports. While recent advances have shown promise, existing medical vision-language methods have several limitations. First, disease information is highly sensitive to spatial relationships, yet conventional data augmentation techniques such as flipping and rotation alter spatial positions. Second, relying solely on global image-text alignment is insufficient to capture localized pathological manifestations that are critical for accurate cross-modal correspondence. Third, global contrastive learning is prone to false negatives since medical images have limited varieties of conditions, increasing the likelihood of similar cases being incorrectly treated as negatives. In this work, we propose MedMALA, a novel MVLP framework that addresses these limitations through two key innovations. First, we develop a Learning Augmentation Network based on VAE, which generates adaptive transformations in the feature space while preserving spatial-semantic correspondences, thereby overcoming the inadequacy of conventional augmentation techniques for medical images. Second, we introduce a Multi-Granularity Alignment mechanism based on contrastive learning, comprising global image-text alignment, anatomy-level spatial alignment, and manifestation-level cross-modal alignment, which enables comprehensive alignment from global to fine-grained semantic levels. Notably, the manifestation-level alignment employs clustering to assign identical pseudo-labels to samples with similar disease manifestations, effectively mitigating false negative issues. We validate MedMALA on four medical benchmarks under both zero-shot and fine-tuning settings, achieving competitive performance that generally outperforms recent approaches in zero-shot scenarios, and state-of-the-art performance in fine-tuning scenarios, particularly in few-shot fine-tuning.