Learning Efficient Feature Adapters for Vision-Language Models
摘要
Contrastive Language-Image Pretraining (CLIP) has demonstrated superior generalization capabilities in image classification over general visual domains. However, for some specific downstream tasks, CLIP still suffers from unsatisfactory results due to semantic gaps. To this end, CoOp has been proposed to adopt learnable prompt tokens for few-shot learning. In this chapter, we showcase an alternative to utilize feature adapters to enhance the few-shot image classification performance of CLIP. We first introduce CLIP-Adapter, an adapter architecture equipped after CLIP’s vision encoder for injecting downstream knowledge. On top of that, we further present two upgraded variants, Tip-Adapter and Tip-Adapter-F, a training-free adaption method, and its fine-tuning variant. They incorporate the few-shot semantics with the pretrained CLIP with a constructed cache model, which further achieve both effectiveness and efficiency for leading image classification performance.