With pre-trained vision-language models, such as CLIP, exhibiting excellent zero-shot generalization capabilities in various downstream tasks. Recently, the application of method prompt tuning to pre-trained vision-language models has become a focal point of attention, showing great potential in vision recognition and transfer learning capabilities over various downstream tasks. However, we find that the existing prompt learning methods, such as, single-modal prompt learning methods CoOp only provide learnable prompts in the text branch, which fails to realize the interaction of multi-modal information between vision and language, resulting in poor generalization ability. For example, multi-modal prompt learning methods MaPLe achieves simple information interactions between the two modalities only through projections between visual and textual prompts, and may not be possible to effectively utilize the information between the two modalities. Therefore, we propose a new method called Dual-text Guided Cascading Visual Prompt (DGCVP). This method uses Cascading Visual Prompt Tuning (CVPT) module to realize the deep fusion of two modalities with the effective participation of vision and language information. We conducted extensive experiments on three benchmarks. Comprehensive experimental analysis shows that DGCVP exhibits better performance compared to previous state-of-the-art prompt learning methods.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Dual-text Guided Cascading Visual Prompt for Vision-Language Models

  • Zebao Zhang,
  • Wenlong Niu,
  • Yue Yang

摘要

With pre-trained vision-language models, such as CLIP, exhibiting excellent zero-shot generalization capabilities in various downstream tasks. Recently, the application of method prompt tuning to pre-trained vision-language models has become a focal point of attention, showing great potential in vision recognition and transfer learning capabilities over various downstream tasks. However, we find that the existing prompt learning methods, such as, single-modal prompt learning methods CoOp only provide learnable prompts in the text branch, which fails to realize the interaction of multi-modal information between vision and language, resulting in poor generalization ability. For example, multi-modal prompt learning methods MaPLe achieves simple information interactions between the two modalities only through projections between visual and textual prompts, and may not be possible to effectively utilize the information between the two modalities. Therefore, we propose a new method called Dual-text Guided Cascading Visual Prompt (DGCVP). This method uses Cascading Visual Prompt Tuning (CVPT) module to realize the deep fusion of two modalities with the effective participation of vision and language information. We conducted extensive experiments on three benchmarks. Comprehensive experimental analysis shows that DGCVP exhibits better performance compared to previous state-of-the-art prompt learning methods.