TriVSS-Net: Visual, Spatial, and Semantic Fusion Transformer for Two-View Correspondence Learning
摘要
Two-view correspondence learning aims to establish accurate matches between similar objects across images. Existing approaches primarily rely on spatial information while neglecting the guidance of scene context and semantic understanding, which are essential for image matching in challenging conditions. To address this limitation, we present TriVSS-Net, a novel Transformer-based framework that integrates Visual, Spatial, and Semantic information for correspondence learning. TriVSS-Net employs three dedicated feature extractors: a local visual module that captures patch-level features around keypoints, a Vision Transformer (ViT) backbone for semantic embedding extraction, and a spatial encoder that models geometric relationships among initial matches. For effective cross-modal fusion, we introduce the TriVSS Integrator, which leverages multi-head self-attention to dynamically align and combine the three feature types. Finally, our proposed Consistency Learning Transformer (CLT) module refines the matches through graph-based propagation with local-global context enhancement, effectively filtering outliers while preserving geometrically consistent correspondences. We evaluate TriVSS-Net through extensive ablation studies and comparative experiments on the camera pose estimation task. The results demonstrate TriVSS-Net’s superiority over state-of-the-art methods on both indoor and outdoor benchmarks.