Dual Consistency Matching for Semi-Supervised Semantic Correspondence
摘要
Establishing correspondences across images sharing the same category remains a challenging task, primarily due to the large intra-class variations and the presence of background clutter. Typically, addressing these challenges necessitates an extensive amount of manually labeled data. However, pixel-level labeling is both time-consuming and labor-intensive. In this paper, we propose a novel teacher-student framework for semi-supervised semantic correspondence, termed Dual Consistency Matching (DCM). We introduce neighborhood shift consistency and semantic consistency to generate reliable pseudo labels, ensuring geometric and semantic coherence, respectively. Unlike previous methods relying on nearest neighbor search at the pixel level, our neighborhood shift consistency enables subpixel accurate estimation through offsets. Semantic consistency, on the other hand, aims to filter out matches by discarding matches that lack semantic coherence despite geometric consistency. Additionally, we propose part-aware prototype learning to impose spatial constraints through the identification of key parts of the object. These filtering strategies enhance the quality of pseudo labels generated by the teacher model. Our framework leverages a pre-trained large-scale vision model as the backbone, which is fine-tuned to improve its representation capabilities. Extensive experiments have been conducted to demonstrate the effectiveness of our proposed method. It is worth noting that our proposed DCM achieves 79.6% PCK@0.1 on the SPair-71k dataset using only 1% labeled data, with an average of 29 training samples per category. Moreover, if 5% labeled data is employed, our method obtains a significantly higher PCK@0.1 score of 86.9%, even surpassing the performance of a fully supervised model trained with all labeled data.