Scholarlink

<p>Composed Image Retrieval (CIR) retrieves a target image similar to a reference image, guided by a provided textual modification (<i>i.e.</i>, a triplet with&lt;reference image, text, target image&gt;). Previous works on CIR can largely be developed into two categories: supervised learning approaches and weakly supervised (<i>i.e.</i>, zero-shot) learning approaches. Supervised learning CIR models require labeled triplets which may not be easily obtained and limit the widespread use of CIR and its scalability. On the other hand, a weakly supervised learning approach (also called zero-shot CIR), can be relatively easily trained with image-caption pairs without considering the image-to-image relation (<i>i.e.</i>, no supervised triplet required), but this approach tends to yield lower accuracy. In this paper, we extend the application of existing Composed Image Retrieval (CIR) into semi-supervised learning, domain adaptation, and test-time adaptation contexts by exploiting only unlabeled image data. Previous approaches cannot be directly applied to these settings, as it is not trivial to leverage fully unlabeled data for CIR. To address this, we propose a new approach and settings where we identify a reference image and its associated target images in auxiliary image data. Our method involves training a large language model-based Visual Delta Generator (VDG) to produce textual descriptions of the visual differences (<i>i.e.</i>, visual deltas) between these images. VDG, equipped with fluent language knowledge and being model agnostic, can generate pseudo-triplets to boost the performance of CIR models in diverse settings including semi-supervised CIR, domain adaptation, and test-time adaptation. Our approach significantly not only improves the existing supervised learning approaches and achieves state-of-the-art results on the CIR benchmarks but also expands the application of CIR across diverse settings.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Visual delta generation with large multi-modal models enhances composed image retrieval using unlabeled data

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年7月28日

查看全文
  • Young Kyun Jang,
  • Donghyun Kim

摘要

Composed Image Retrieval (CIR) retrieves a target image similar to a reference image, guided by a provided textual modification (i.e., a triplet with<reference image, text, target image>). Previous works on CIR can largely be developed into two categories: supervised learning approaches and weakly supervised (i.e., zero-shot) learning approaches. Supervised learning CIR models require labeled triplets which may not be easily obtained and limit the widespread use of CIR and its scalability. On the other hand, a weakly supervised learning approach (also called zero-shot CIR), can be relatively easily trained with image-caption pairs without considering the image-to-image relation (i.e., no supervised triplet required), but this approach tends to yield lower accuracy. In this paper, we extend the application of existing Composed Image Retrieval (CIR) into semi-supervised learning, domain adaptation, and test-time adaptation contexts by exploiting only unlabeled image data. Previous approaches cannot be directly applied to these settings, as it is not trivial to leverage fully unlabeled data for CIR. To address this, we propose a new approach and settings where we identify a reference image and its associated target images in auxiliary image data. Our method involves training a large language model-based Visual Delta Generator (VDG) to produce textual descriptions of the visual differences (i.e., visual deltas) between these images. VDG, equipped with fluent language knowledge and being model agnostic, can generate pseudo-triplets to boost the performance of CIR models in diverse settings including semi-supervised CIR, domain adaptation, and test-time adaptation. Our approach significantly not only improves the existing supervised learning approaches and achieves state-of-the-art results on the CIR benchmarks but also expands the application of CIR across diverse settings.