<p>A primary challenge in image-text retrieval lies in bridging the heterogeneity gap between visual and textual modalities. Most existing methods address this gap by learning alignments from superficial relationships, yet fail to explicitly model the underlying implicit information within and across modalities. This reliance on surface-level cues leads to semantic ambiguity and an incomplete understanding of content. To address these limitations, we propose SCHN, a Semantic Clarity Enhancement and Cluster-Assisted Learning Hard Alignment Network for image-text retrieval. Our approach: (1) designing a Semantic Clarity Enhancer (SCE) that combines MLPs and stacked self-attention layers with residual connections to extract deep contextual information from local features, thereby enhancing semantic clarity and reducing semantic ambiguity; (2) designing a Cluster-Assisted Learning (CAL) module that employs cluster-based cross-prediction and contrastive learning in a shared prototype space to strengthen fine-grained cross-modal correspondences and learn conceptual prototypes that capture cross-modal semantic commonalities. The results of extensive experiments on the Flickr30k and MS-COCO benchmarks demonstrate that our proposed method achieves superior performance. Here, we show that enhancing intra-modal contextual clarity and learning at a conceptual level are crucial for accurate fine-grained alignment in ITR. Our code is available at <a href="https://github.com/BxBYL/SCHN">https://github.com/BxBYL/SCHN</a>.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Image-text Retrieval via Semantic Clarity Enhancement and Cluster-Assisted Learning

  • Hai Su,
  • Binyan Li,
  • Xiaoming Hu,
  • Hongsong Wang,
  • Dan Xiang

摘要

A primary challenge in image-text retrieval lies in bridging the heterogeneity gap between visual and textual modalities. Most existing methods address this gap by learning alignments from superficial relationships, yet fail to explicitly model the underlying implicit information within and across modalities. This reliance on surface-level cues leads to semantic ambiguity and an incomplete understanding of content. To address these limitations, we propose SCHN, a Semantic Clarity Enhancement and Cluster-Assisted Learning Hard Alignment Network for image-text retrieval. Our approach: (1) designing a Semantic Clarity Enhancer (SCE) that combines MLPs and stacked self-attention layers with residual connections to extract deep contextual information from local features, thereby enhancing semantic clarity and reducing semantic ambiguity; (2) designing a Cluster-Assisted Learning (CAL) module that employs cluster-based cross-prediction and contrastive learning in a shared prototype space to strengthen fine-grained cross-modal correspondences and learn conceptual prototypes that capture cross-modal semantic commonalities. The results of extensive experiments on the Flickr30k and MS-COCO benchmarks demonstrate that our proposed method achieves superior performance. Here, we show that enhancing intra-modal contextual clarity and learning at a conceptual level are crucial for accurate fine-grained alignment in ITR. Our code is available at https://github.com/BxBYL/SCHN.