<p>Graph structures can represent rich semantic relationships and be applied in image-text matching tasks. However, complex graph structures may be susceptible to environmental-specific biases, leading to overfitting and low generalization. In addition, there is still room for improvement of feature learning in existing image-text matching methods. For these issues, we innovatively introduced causal reasoning on graphs and proposed Neighborhood-Interference Independent Graph Mechanisms (NIIGM) to effectively extract the modality variations and collect the corresponding semantics of cross-modal features. Specifically, based on relationships graph analysis, we introduce individual-level neighborhood-interference by intervening in the edges and nodes to explore in-depth causality beneath superficial correlations. The independent graph mechanism modularizes various functions to prevent non-generalizing effects caused by language shortcut bias. In addition, we scrutinized two similarity representations and proposed targeted learning processes for global features and local features. Comprehensive experiments showed that our method can outperform the SoTA image-text matching methods. Adequate ablation experiments validated the effectiveness of each component.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Neighborhood-interference independent graph mechanisms for image-text matching

  • Bo Li,
  • Zhixin Li,
  • You Wu,
  • Xing Wei

摘要

Graph structures can represent rich semantic relationships and be applied in image-text matching tasks. However, complex graph structures may be susceptible to environmental-specific biases, leading to overfitting and low generalization. In addition, there is still room for improvement of feature learning in existing image-text matching methods. For these issues, we innovatively introduced causal reasoning on graphs and proposed Neighborhood-Interference Independent Graph Mechanisms (NIIGM) to effectively extract the modality variations and collect the corresponding semantics of cross-modal features. Specifically, based on relationships graph analysis, we introduce individual-level neighborhood-interference by intervening in the edges and nodes to explore in-depth causality beneath superficial correlations. The independent graph mechanism modularizes various functions to prevent non-generalizing effects caused by language shortcut bias. In addition, we scrutinized two similarity representations and proposed targeted learning processes for global features and local features. Comprehensive experiments showed that our method can outperform the SoTA image-text matching methods. Adequate ablation experiments validated the effectiveness of each component.