Two-view correspondence learning aims to establish accurate matches between similar objects across images. Existing approaches primarily rely on spatial information while neglecting the guidance of scene context and semantic understanding, which are essential for image matching in challenging conditions. To address this limitation, we present TriVSS-Net, a novel Transformer-based framework that integrates Visual, Spatial, and Semantic information for correspondence learning. TriVSS-Net employs three dedicated feature extractors: a local visual module that captures patch-level features around keypoints, a Vision Transformer (ViT) backbone for semantic embedding extraction, and a spatial encoder that models geometric relationships among initial matches. For effective cross-modal fusion, we introduce the TriVSS Integrator, which leverages multi-head self-attention to dynamically align and combine the three feature types. Finally, our proposed Consistency Learning Transformer (CLT) module refines the matches through graph-based propagation with local-global context enhancement, effectively filtering outliers while preserving geometrically consistent correspondences. We evaluate TriVSS-Net through extensive ablation studies and comparative experiments on the camera pose estimation task. The results demonstrate TriVSS-Net’s superiority over state-of-the-art methods on both indoor and outdoor benchmarks.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

TriVSS-Net: Visual, Spatial, and Semantic Fusion Transformer for Two-View Correspondence Learning

  • Wenfei Li,
  • Weiwei Zhou,
  • Yizhang Liu,
  • Shengjie Zhao

摘要

Two-view correspondence learning aims to establish accurate matches between similar objects across images. Existing approaches primarily rely on spatial information while neglecting the guidance of scene context and semantic understanding, which are essential for image matching in challenging conditions. To address this limitation, we present TriVSS-Net, a novel Transformer-based framework that integrates Visual, Spatial, and Semantic information for correspondence learning. TriVSS-Net employs three dedicated feature extractors: a local visual module that captures patch-level features around keypoints, a Vision Transformer (ViT) backbone for semantic embedding extraction, and a spatial encoder that models geometric relationships among initial matches. For effective cross-modal fusion, we introduce the TriVSS Integrator, which leverages multi-head self-attention to dynamically align and combine the three feature types. Finally, our proposed Consistency Learning Transformer (CLT) module refines the matches through graph-based propagation with local-global context enhancement, effectively filtering outliers while preserving geometrically consistent correspondences. We evaluate TriVSS-Net through extensive ablation studies and comparative experiments on the camera pose estimation task. The results demonstrate TriVSS-Net’s superiority over state-of-the-art methods on both indoor and outdoor benchmarks.