This paper presents an approach to enhance continual learning in vision-language models through shared adapters and contrastive learning. Our method leverages a frozen, pre-trained Vision Transformer (ViT) model, while only updating the trainable adapters. To address catastrophic forgetting, we utilize online prototypes—a lightweight representation for storing class information instead of maintaining a traditional memory bank of previous task data. The shared adapter architecture is integrated with contrastive learning, which helps improve model plasticity and reinforces class boundaries without requiring access to previously learned data. Experimental results on ImageNet-A, ImageNet-R, and CIFAR100 datasets demonstrate that our approach achieves competitive performance compared to some state-of-the-art methods on the last score, with accuracy improvements of up to 62.74% on ImageNet-A and 91.93% on CIFAR100 for the previous task, while maintaining a competitive average performance across all tasks.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Enhancing Continual Learning in Vision-Language Models Using Shared Adapter Integrated with Contrastive Learning

  • Quynh-Trang Pham Thi,
  • An Duc Vinh Pham,
  • Tri-Thanh Nguyen,
  • Thanh-Hai Dang

摘要

This paper presents an approach to enhance continual learning in vision-language models through shared adapters and contrastive learning. Our method leverages a frozen, pre-trained Vision Transformer (ViT) model, while only updating the trainable adapters. To address catastrophic forgetting, we utilize online prototypes—a lightweight representation for storing class information instead of maintaining a traditional memory bank of previous task data. The shared adapter architecture is integrated with contrastive learning, which helps improve model plasticity and reinforces class boundaries without requiring access to previously learned data. Experimental results on ImageNet-A, ImageNet-R, and CIFAR100 datasets demonstrate that our approach achieves competitive performance compared to some state-of-the-art methods on the last score, with accuracy improvements of up to 62.74% on ImageNet-A and 91.93% on CIFAR100 for the previous task, while maintaining a competitive average performance across all tasks.