The aim of multimodal intent detection is to understand user intent through multimodal data. Currently, there are three main challenges in this task. First, modality-specific information is easily lost during fusion. Second, insufficient cross-modal interaction leads to limited synergy. The representation space lacks discriminative power, making it difficult to achieve intraclass compactness and interclass separability. To address these issues, we propose a novel framework called HMTC, which uniquely combines customized hierarchical modality representation learning and triplet-based contrastive learning, both tailored for the intent detection task. This approach overcomes previous limitations by effectively preserving modality-specific information, enhancing cross-modal synergy, and improving discriminability in the embedding space. Hierarchical modality representation learning captures unique intramodal features, shared cross-modal information, and local interactions between modality pairs through dual modeling (MSE and GSE) and the pairwise synergistic encoder (PSE). It ensures information integrity and cross-modal consistency using reconstruction constraints and triplet alignment loss. Triplet-based contrastive representation learning introduces triplet contrastive loss to enhance intraclass compactness and interclass separability in the global intent embedding space. It compensates for the shortcomings of traditional classification loss in embedding space discrimination and improves the model’s generalization ability. Comprehensive experiments on two datasets demonstrate that our framework outperforms multiple baseline methods.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Improving Intent Detection with Hierarchical Multimodal Representation and Triplet Contrastive Learning

  • Lanlan Lu,
  • Qimeng Yang,
  • Yi Liu

摘要

The aim of multimodal intent detection is to understand user intent through multimodal data. Currently, there are three main challenges in this task. First, modality-specific information is easily lost during fusion. Second, insufficient cross-modal interaction leads to limited synergy. The representation space lacks discriminative power, making it difficult to achieve intraclass compactness and interclass separability. To address these issues, we propose a novel framework called HMTC, which uniquely combines customized hierarchical modality representation learning and triplet-based contrastive learning, both tailored for the intent detection task. This approach overcomes previous limitations by effectively preserving modality-specific information, enhancing cross-modal synergy, and improving discriminability in the embedding space. Hierarchical modality representation learning captures unique intramodal features, shared cross-modal information, and local interactions between modality pairs through dual modeling (MSE and GSE) and the pairwise synergistic encoder (PSE). It ensures information integrity and cross-modal consistency using reconstruction constraints and triplet alignment loss. Triplet-based contrastive representation learning introduces triplet contrastive loss to enhance intraclass compactness and interclass separability in the global intent embedding space. It compensates for the shortcomings of traditional classification loss in embedding space discrimination and improves the model’s generalization ability. Comprehensive experiments on two datasets demonstrate that our framework outperforms multiple baseline methods.