In previous work, multimodal sentiment analysis(MSA) has achieved great success in the multimedia field. However, in real-world application, most multimodal data are uncertain and incomplete. In this case, the effect of MSA with complete modalities cannot be guaranteed. In this work, we proposed a cascade residual autoencoder for MSA with Uncertain missing modalities(CRAM). This model can perform well in the face of randomly masked multimodal data by learning robust jount multimodal representations. In order to optimize the reconstructed joint modality representation, a supervised contrastive learning algorithm optimized under small batches is proposed here. Our model has good performance on both CMU-MOSI and CMU-MOSEI datasets, verifying the superiority of our method in the missing multimodal scenario.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Reconstruction Based Multimodal Sentiment Analysis with Uncertain Missing Modalities

  • Yi Zhou,
  • Yong Hao,
  • Bo Li

摘要

In previous work, multimodal sentiment analysis(MSA) has achieved great success in the multimedia field. However, in real-world application, most multimodal data are uncertain and incomplete. In this case, the effect of MSA with complete modalities cannot be guaranteed. In this work, we proposed a cascade residual autoencoder for MSA with Uncertain missing modalities(CRAM). This model can perform well in the face of randomly masked multimodal data by learning robust jount multimodal representations. In order to optimize the reconstructed joint modality representation, a supervised contrastive learning algorithm optimized under small batches is proposed here. Our model has good performance on both CMU-MOSI and CMU-MOSEI datasets, verifying the superiority of our method in the missing multimodal scenario.