Scholarlink

<p>Foundation models pre-trained on large-scale vision-language data, such as CLIP, exhibit strong zero-shot visual recognition but suffer from catastrophic forgetting when fine-tuned on sequential task streams. We introduce the Hierarchical Prompt Composition Network (HPC-NET), a parameter-efficient architecture for continual learning in open-world environments. HPC-NET decomposes learnable prompts into three tiers: <i>foundational prompts</i> that encode broad visual primitives, <i>compositional prompts</i> that aggregate mid-level patterns, and <i>instance prompts</i> that capture category-specific features. A <i>Semantic Prototype Anchoring</i> (SPA) mechanism constrains each tier to stable CLIP reference embeddings, mitigating semantic drift. A <i>Contrastive Prompt Router</i> (CPR) dynamically selects a sparse subset of prompts per input via attention-based routing. Across five benchmarks–Split-CIFAR100, Split-ImageNet-R, CORe50, MedStream-7k, and Split-CUB200–HPC-NET achieves <InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(84.3 \pm 0.9\%\)</EquationSource> </InlineEquation> average accuracy, a 5.4 percentage-point improvement over the strongest baseline, while retaining <InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(98.4\%\)</EquationSource> </InlineEquation> of the base model’s zero-shot performance and requiring only 2.1 M additional parameters, compared to 24.3 M for adapter-based methods.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Hierarchical prompt composition for memory-efficient open-world continual learning in vision-language foundation models

  • 期刊论文

  • 发表日期：

    2026年7月9日

查看全文
  • Siham Rebbah

摘要

Foundation models pre-trained on large-scale vision-language data, such as CLIP, exhibit strong zero-shot visual recognition but suffer from catastrophic forgetting when fine-tuned on sequential task streams. We introduce the Hierarchical Prompt Composition Network (HPC-NET), a parameter-efficient architecture for continual learning in open-world environments. HPC-NET decomposes learnable prompts into three tiers: foundational prompts that encode broad visual primitives, compositional prompts that aggregate mid-level patterns, and instance prompts that capture category-specific features. A Semantic Prototype Anchoring (SPA) mechanism constrains each tier to stable CLIP reference embeddings, mitigating semantic drift. A Contrastive Prompt Router (CPR) dynamically selects a sparse subset of prompts per input via attention-based routing. Across five benchmarks–Split-CIFAR100, Split-ImageNet-R, CORe50, MedStream-7k, and Split-CUB200–HPC-NET achieves \(84.3 \pm 0.9\%\) average accuracy, a 5.4 percentage-point improvement over the strongest baseline, while retaining \(98.4\%\) of the base model’s zero-shot performance and requiring only 2.1 M additional parameters, compared to 24.3 M for adapter-based methods.