Improving Intent Detection with Hierarchical Multimodal Representation and Triplet Contrastive Learning
摘要
The aim of multimodal intent detection is to understand user intent through multimodal data. Currently, there are three main challenges in this task. First, modality-specific information is easily lost during fusion. Second, insufficient cross-modal interaction leads to limited synergy. The representation space lacks discriminative power, making it difficult to achieve intraclass compactness and interclass separability. To address these issues, we propose a novel framework called HMTC, which uniquely combines customized hierarchical modality representation learning and triplet-based contrastive learning, both tailored for the intent detection task. This approach overcomes previous limitations by effectively preserving modality-specific information, enhancing cross-modal synergy, and improving discriminability in the embedding space. Hierarchical modality representation learning captures unique intramodal features, shared cross-modal information, and local interactions between modality pairs through dual modeling (MSE and GSE) and the pairwise synergistic encoder (PSE). It ensures information integrity and cross-modal consistency using reconstruction constraints and triplet alignment loss. Triplet-based contrastive representation learning introduces triplet contrastive loss to enhance intraclass compactness and interclass separability in the global intent embedding space. It compensates for the shortcomings of traditional classification loss in embedding space discrimination and improves the model’s generalization ability. Comprehensive experiments on two datasets demonstrate that our framework outperforms multiple baseline methods.