Dual-text Guided Cascading Visual Prompt for Vision-Language Models
摘要
With pre-trained vision-language models, such as CLIP, exhibiting excellent zero-shot generalization capabilities in various downstream tasks. Recently, the application of method prompt tuning to pre-trained vision-language models has become a focal point of attention, showing great potential in vision recognition and transfer learning capabilities over various downstream tasks. However, we find that the existing prompt learning methods, such as, single-modal prompt learning methods CoOp only provide learnable prompts in the text branch, which fails to realize the interaction of multi-modal information between vision and language, resulting in poor generalization ability. For example, multi-modal prompt learning methods MaPLe achieves simple information interactions between the two modalities only through projections between visual and textual prompts, and may not be possible to effectively utilize the information between the two modalities. Therefore, we propose a new method called Dual-text Guided Cascading Visual Prompt (DGCVP). This method uses Cascading Visual Prompt Tuning (CVPT) module to realize the deep fusion of two modalities with the effective participation of vision and language information. We conducted extensive experiments on three benchmarks. Comprehensive experimental analysis shows that DGCVP exhibits better performance compared to previous state-of-the-art prompt learning methods.