<p>The image-text retrieval task aims to build a semantic bridge between images and text. The primary challenge lies in accurately measuring image-text similarity. Existing methods depend on pre-trained object detectors to extract image region features and Transformers for text processing. This approach often results in inconsistent semantic distribution during cross-modal alignment. In addition, there are significant differences in the representation forms and distribution of images and text. In order to accurately represent the real world, images contain a large amount of detail and background information, resulting in higher information redundancy than text. This redundant information not only increases the computational burden but also tends to cause inaccurate alignment issues. To address these issues, this paper proposes a Text Semantic-Guided Adaptive Feature Aggregation (TGAA) framework based on Transformer architecture. The framework designs a text semantic-guided feature purification module, which leverages both text supervision and image self-supervision to guide the filtering of redundant image features, thereby enhancing the compactness and discriminability of image representations. An adaptive feature aggregation strategy is then employed to further enhance the expressiveness of global features without increasing architectural complexity. The model conducted extensive experiments on two benchmark datasets, Flickr30k and MS COCO. It achieved a maximum improvement of 29.8% in the RSUM compared to the baseline. These results highlight the effectiveness of the proposed framework and its potential in cross-modal retrieval tasks, confirming its competitive retrieval performance.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Text semantic-guided adaptive feature aggregation for image-text retrieval

  • Yajie Gu,
  • Mingjie Wang,
  • Jianhou Gan,
  • Yiming Zhao,
  • Jiatian Mei,
  • Chuanzhi Zhang

摘要

The image-text retrieval task aims to build a semantic bridge between images and text. The primary challenge lies in accurately measuring image-text similarity. Existing methods depend on pre-trained object detectors to extract image region features and Transformers for text processing. This approach often results in inconsistent semantic distribution during cross-modal alignment. In addition, there are significant differences in the representation forms and distribution of images and text. In order to accurately represent the real world, images contain a large amount of detail and background information, resulting in higher information redundancy than text. This redundant information not only increases the computational burden but also tends to cause inaccurate alignment issues. To address these issues, this paper proposes a Text Semantic-Guided Adaptive Feature Aggregation (TGAA) framework based on Transformer architecture. The framework designs a text semantic-guided feature purification module, which leverages both text supervision and image self-supervision to guide the filtering of redundant image features, thereby enhancing the compactness and discriminability of image representations. An adaptive feature aggregation strategy is then employed to further enhance the expressiveness of global features without increasing architectural complexity. The model conducted extensive experiments on two benchmark datasets, Flickr30k and MS COCO. It achieved a maximum improvement of 29.8% in the RSUM compared to the baseline. These results highlight the effectiveness of the proposed framework and its potential in cross-modal retrieval tasks, confirming its competitive retrieval performance.