DiSCo: Disrupting Semantic Consistency for Transferable Cross-Modal Adversarial Attacks
摘要
Current vision-language pretraining (VLP) models exhibit significant vulnerability to adversarial examples. Compared to white-box attacks, transfer attacks on unseen models better reflect real-world scenarios and offer greater research value. However, existing methods generally fail to effectively disrupt vision-language semantic alignment mechanism, limiting the transferability of adversarial examples. To address this issue, we propose DiSCo, a unified adversarial example generation framework. Specifically, we propose a perturbation generation strategy that disrupts semantic topological structures and image–text feature similarity, introducing cross-modal inconsistency that degrades downstream task performance. To mitigate the influence of redundant feature dimensions, we project image-text features into a semantic subspace constructed via clustering and hierarchical sampling before optimizing the adversarial trajectory. Furthermore, we design a cross-modal attention-based perturbation modulation mechanism that identifies text-relevant image regions and adjusts perturbations accordingly. Extensive experiments across various downstream tasks and VLP architectures demonstrate that our method significantly outperforms state-of-the-art approaches, with superior transferability and generalization.