<p>For large-scale cross-modal retrieval task, unsupervised cross-modal hashing has attracted considerable attention due to its efficient retrieval performance and label irrelevance. However, existing unsupervised cross-modal hashing methods only focus on whether specific semantic content exists in different modalities, while neglecting the relative semantic relationship. To address the above issue, we propose a novel Relative Semantic Relationship Preserving Hashing (RSRPH), which integrates the feature fusion technique and self-attention mechanism to capture the intra- and inter-modal relative semantic relationships. In the Feature Fusion Attention (FFA) module, the transformer model learns the fusion features to represent the relative semantic relationship, which can minimize differences among the relative weight values of similar semantic features. To enhance the semantic consistency among different modal samples with similar content, the Fusion Contrastive Learning (FCL) module maximizes the mutual information between similar image and text samples. FCL module ensures that the hash codes of similar samples are closer in the Hamming space. The Similarity Consistency Loss (SCL) module minimizes the discrepancy between the hash similarity matrix and the original feature similarity matrix, which makes the intra- and inter-modal fusion relative semantic relationships be consistent with each other. We conduct the cross-modal retrieval comparative experiments on three widely used datasets MIRFLICKR-25K, NUS-WIDE and MS COCO. The experimental results demonstrate that the proposed RSRPH method achieves the superior retrieval performance than the state-of-the-art methods.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Relative semantic relationship preserving hashing for unsupervised cross-modal retrieval

  • Limeng Gao,
  • Zhen Wang,
  • Xinzhong Wang,
  • Zhen Zheng,
  • Chenchen Liu

摘要

For large-scale cross-modal retrieval task, unsupervised cross-modal hashing has attracted considerable attention due to its efficient retrieval performance and label irrelevance. However, existing unsupervised cross-modal hashing methods only focus on whether specific semantic content exists in different modalities, while neglecting the relative semantic relationship. To address the above issue, we propose a novel Relative Semantic Relationship Preserving Hashing (RSRPH), which integrates the feature fusion technique and self-attention mechanism to capture the intra- and inter-modal relative semantic relationships. In the Feature Fusion Attention (FFA) module, the transformer model learns the fusion features to represent the relative semantic relationship, which can minimize differences among the relative weight values of similar semantic features. To enhance the semantic consistency among different modal samples with similar content, the Fusion Contrastive Learning (FCL) module maximizes the mutual information between similar image and text samples. FCL module ensures that the hash codes of similar samples are closer in the Hamming space. The Similarity Consistency Loss (SCL) module minimizes the discrepancy between the hash similarity matrix and the original feature similarity matrix, which makes the intra- and inter-modal fusion relative semantic relationships be consistent with each other. We conduct the cross-modal retrieval comparative experiments on three widely used datasets MIRFLICKR-25K, NUS-WIDE and MS COCO. The experimental results demonstrate that the proposed RSRPH method achieves the superior retrieval performance than the state-of-the-art methods.