Neighborhood-interference independent graph mechanisms for image-text matching
摘要
Graph structures can represent rich semantic relationships and be applied in image-text matching tasks. However, complex graph structures may be susceptible to environmental-specific biases, leading to overfitting and low generalization. In addition, there is still room for improvement of feature learning in existing image-text matching methods. For these issues, we innovatively introduced causal reasoning on graphs and proposed Neighborhood-Interference Independent Graph Mechanisms (NIIGM) to effectively extract the modality variations and collect the corresponding semantics of cross-modal features. Specifically, based on relationships graph analysis, we introduce individual-level neighborhood-interference by intervening in the edges and nodes to explore in-depth causality beneath superficial correlations. The independent graph mechanism modularizes various functions to prevent non-generalizing effects caused by language shortcut bias. In addition, we scrutinized two similarity representations and proposed targeted learning processes for global features and local features. Comprehensive experiments showed that our method can outperform the SoTA image-text matching methods. Adequate ablation experiments validated the effectiveness of each component.