<p>Existing approaches for multimodal hand feature fusion typically involve extracting features from different modalities independently and subsequently merging them directly. However, these methods overlook potential correlations between modalities, leading to limitations in effectively leveraging complementary information across modalities. Therefore, this paper proposes a multimodal hand feature fusion method based on modality-related feature interaction (MR-FIFM). Firstly, a specific-shared feature extraction network(SSFEN) is constructed, preserving the integrity of features of each modality while enhancing the correlations between modalities. Secondly, a class-constrained multimodal metric loss is proposed to avoid the gradient vanishing problem and enhance the category representation capability of features. Finally, a feature interaction fusion module(FIFM) is proposed, allowing the model to simultaneously focus on features at different positions within each modality to enhance intra-modality features. Simultaneously, the features between the two modalities are interacted, enabling the model to dynamically adjust its own feature representation based on the features of the other modality. Experimental results on public datasets, including palmprint + palm-vein (PolyU_Palmprint+PolyU_NIR, PolyU_Blue+PolyU_NIR), and fingerprint + finger-vein (NUPT_FP+NUPT_FV), demonstrate that the proposed method outperforms existing multimodal fusion recognition methods for hand features.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multimodal hand feature recognition base on modality-related feature interaction

  • Huabin Wang,
  • Li Zhang,
  • Shicheng Wei,
  • Fei Liu,
  • Liang Tao

摘要

Existing approaches for multimodal hand feature fusion typically involve extracting features from different modalities independently and subsequently merging them directly. However, these methods overlook potential correlations between modalities, leading to limitations in effectively leveraging complementary information across modalities. Therefore, this paper proposes a multimodal hand feature fusion method based on modality-related feature interaction (MR-FIFM). Firstly, a specific-shared feature extraction network(SSFEN) is constructed, preserving the integrity of features of each modality while enhancing the correlations between modalities. Secondly, a class-constrained multimodal metric loss is proposed to avoid the gradient vanishing problem and enhance the category representation capability of features. Finally, a feature interaction fusion module(FIFM) is proposed, allowing the model to simultaneously focus on features at different positions within each modality to enhance intra-modality features. Simultaneously, the features between the two modalities are interacted, enabling the model to dynamically adjust its own feature representation based on the features of the other modality. Experimental results on public datasets, including palmprint + palm-vein (PolyU_Palmprint+PolyU_NIR, PolyU_Blue+PolyU_NIR), and fingerprint + finger-vein (NUPT_FP+NUPT_FV), demonstrate that the proposed method outperforms existing multimodal fusion recognition methods for hand features.