Scholarlink

Medical vision-language models (Med-VLMs) have shown strong generalization in downstream tasks by leveraging large-scale image-text pretraining. However, their vulnerability to adversarial perturbations poses critical risks in safety-sensitive applications. While randomized smoothing provides certifiable guarantees against such perturbations, it requires the model to remain accurate under Gaussian noise—an assumption that fails in practice without specialized adaptation. Prior work, such as PromptSmooth, introduced text-only prompt tuning to improve robustness, but neglected visual adaptation and struggled under high noise levels. In this paper, we propose PromptSmooth++, a multimodal adaptation framework that enhances the certified robustness of frozen Med-VLMs under randomized smoothing. Our method introduces two complementary variants: (i) a few-shot strategy that jointly optimizes visual and textual prompts for noise-aware cross-modal alignment, and (ii) a zero-shot approach that performs test-time vision-side adaptation using lightweight Low-Rank Adapters (LoRA), optimized with a self-supervised entropy loss on noisy inputs. Extensive experiments across multiple Med-VLMs and multiple datasets spanning diverse medical modalities demonstrate that PromptSmooth++ significantly improves certified accuracy over existing baselines while maintaining high clean performance and computational efficiency. Our results show that modality-aware prompting and vision-side adaptation are both essential for certifiably robust medical imaging systems. Code is available at https://github.com/fahadshamshad/multimodal-promptsmooth .

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Parameter-Efficient Multimodal Adaptation for Certified Robustness of Medical Vision-Language Models

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Fahad Shamshad,
  • Noor Hussein,
  • Karthik Nandakumar

摘要

Medical vision-language models (Med-VLMs) have shown strong generalization in downstream tasks by leveraging large-scale image-text pretraining. However, their vulnerability to adversarial perturbations poses critical risks in safety-sensitive applications. While randomized smoothing provides certifiable guarantees against such perturbations, it requires the model to remain accurate under Gaussian noise—an assumption that fails in practice without specialized adaptation. Prior work, such as PromptSmooth, introduced text-only prompt tuning to improve robustness, but neglected visual adaptation and struggled under high noise levels. In this paper, we propose PromptSmooth++, a multimodal adaptation framework that enhances the certified robustness of frozen Med-VLMs under randomized smoothing. Our method introduces two complementary variants: (i) a few-shot strategy that jointly optimizes visual and textual prompts for noise-aware cross-modal alignment, and (ii) a zero-shot approach that performs test-time vision-side adaptation using lightweight Low-Rank Adapters (LoRA), optimized with a self-supervised entropy loss on noisy inputs. Extensive experiments across multiple Med-VLMs and multiple datasets spanning diverse medical modalities demonstrate that PromptSmooth++ significantly improves certified accuracy over existing baselines while maintaining high clean performance and computational efficiency. Our results show that modality-aware prompting and vision-side adaptation are both essential for certifiably robust medical imaging systems. Code is available at https://github.com/fahadshamshad/multimodal-promptsmooth .