Scholarlink

Pre-trained visual language models (VLMs) like CLIP excel at cross-modal reasoning but remain vulnerable to adversarial attacks. This paper introduces Adversarial Prompt Increment (API) learning to enhance VLMs’ adversarial robustness. Our approach starts with a base prompt ( \(P_{\text {base}}\) ) optimized on clean data, then adds an adversarially trained residual vector ( \(\varDelta P\) ) to form a robust prompt: \(P_{\text {robust}} = P_{\text {base}} + \varDelta P\) . This design allows \(\varDelta P\) to counter adversarial perturbations while \(P_{\text {base}}\) preserves semantic integrity, achieving robustness without extensive parameter training or architectural changes. To enhance generalization, we use dynamic parameter tuning by varying perturbation budgets during training. Experiments on 8 benchmark datasets show that our approach improves clean accuracy by +1.64% and adversarial robustness by +1.42% over baselines under PGD attacks ( \(\varepsilon = 4/255\) ), with notable effectiveness also demonstrated at ( \(\varepsilon = 1/255\) ).

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Adversarial Prompt Increment for Robust Vision-Language Models

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Zhouchen Yang,
  • Zhongchen Ma

摘要

Pre-trained visual language models (VLMs) like CLIP excel at cross-modal reasoning but remain vulnerable to adversarial attacks. This paper introduces Adversarial Prompt Increment (API) learning to enhance VLMs’ adversarial robustness. Our approach starts with a base prompt ( \(P_{\text {base}}\) ) optimized on clean data, then adds an adversarially trained residual vector ( \(\varDelta P\) ) to form a robust prompt: \(P_{\text {robust}} = P_{\text {base}} + \varDelta P\) . This design allows \(\varDelta P\) to counter adversarial perturbations while \(P_{\text {base}}\) preserves semantic integrity, achieving robustness without extensive parameter training or architectural changes. To enhance generalization, we use dynamic parameter tuning by varying perturbation budgets during training. Experiments on 8 benchmark datasets show that our approach improves clean accuracy by +1.64% and adversarial robustness by +1.42% over baselines under PGD attacks ( \(\varepsilon = 4/255\) ), with notable effectiveness also demonstrated at ( \(\varepsilon = 1/255\) ).