Generalized zero-shot enables the recognition of unseen classes by transferring semantic knowledge from seen classes, which is achieved by learning the latent relationship between visual features and attribute features. Previous works simply align the feature of an image with its associated attribute descriptions. However, direct alignment of image and text modalities often results in spurious visual-semantic correlations, owing to the inherent differences in semantic levels and granularity between the features extracted from image samples and text representations. To solve this situation, we introduce a Finite Discrete Token Module (FDTM) across the two modalities, using unified token representations to encode images and texts, achieving consistency between image and text embeddings at both the granularity and semantic levels. In addition, we propose a Semantic Correlation Intervention Module (SCIM) analyzes the relationships between different attribute response values in the attention mechanism, making the predictions closer to the ground truth and improving attribute prediction accuracy. The proposed method is evaluated on three ZSL benchmarks, and the results demonstrate the feasibility of our proposed method.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Unified Token Representation and Accurate Attribute Prediction for Generalized Zero-Shot Learning

  • Rui Ren,
  • Huihui Bai

摘要

Generalized zero-shot enables the recognition of unseen classes by transferring semantic knowledge from seen classes, which is achieved by learning the latent relationship between visual features and attribute features. Previous works simply align the feature of an image with its associated attribute descriptions. However, direct alignment of image and text modalities often results in spurious visual-semantic correlations, owing to the inherent differences in semantic levels and granularity between the features extracted from image samples and text representations. To solve this situation, we introduce a Finite Discrete Token Module (FDTM) across the two modalities, using unified token representations to encode images and texts, achieving consistency between image and text embeddings at both the granularity and semantic levels. In addition, we propose a Semantic Correlation Intervention Module (SCIM) analyzes the relationships between different attribute response values in the attention mechanism, making the predictions closer to the ground truth and improving attribute prediction accuracy. The proposed method is evaluated on three ZSL benchmarks, and the results demonstrate the feasibility of our proposed method.