The advancement of artificial intelligence in multimodal learning has driven the need for effective representation techniques that capture dependencies. This research presents Multimodal Representation with Variational AutoEncoder(MRVAE), a novel framework that leverages variational autoencoders to learn structured latent representations from image and text embeddings. Unlike deterministic models, MRVAE employs stochastic latent modeling, enabling better feature disentanglement and semantic alignment. The framework integrates convolutional neural networks for hierarchical image feature extraction and a variational autoencoder for encoding text embeddings, ensuring a probabilistic latent space. These learned features are fused into a joint representation, which is utilized for classification. MRVAE is evaluated against deterministic multimodal autoencoders such as Multimodal Representation with AutoEncoder(MRAE) and baseline models like Multimodal Encoder Decoder Architecture-I(MMEDA-I) in a book rating classification task, where it achieves 97.85% accuracy, outperforming MRAE (96.50% accuracy) and MMEDA-I (68.60% accuracy). These findings position MRVAE as a promising framework for multimodal learning tasks, particularly in applications requiring joint image-text understanding, reinforcing the benefits of variational modeling in structured representation learning.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Enhancing Classification with Joint Representation Learning on Multimodal Data

  • Neha Dhirendra Sirur,
  • Padmashree Desai,
  • Sujatha C,
  • Uma Mudengudi,
  • Ramesh Ashok Tabib

摘要

The advancement of artificial intelligence in multimodal learning has driven the need for effective representation techniques that capture dependencies. This research presents Multimodal Representation with Variational AutoEncoder(MRVAE), a novel framework that leverages variational autoencoders to learn structured latent representations from image and text embeddings. Unlike deterministic models, MRVAE employs stochastic latent modeling, enabling better feature disentanglement and semantic alignment. The framework integrates convolutional neural networks for hierarchical image feature extraction and a variational autoencoder for encoding text embeddings, ensuring a probabilistic latent space. These learned features are fused into a joint representation, which is utilized for classification. MRVAE is evaluated against deterministic multimodal autoencoders such as Multimodal Representation with AutoEncoder(MRAE) and baseline models like Multimodal Encoder Decoder Architecture-I(MMEDA-I) in a book rating classification task, where it achieves 97.85% accuracy, outperforming MRAE (96.50% accuracy) and MMEDA-I (68.60% accuracy). These findings position MRVAE as a promising framework for multimodal learning tasks, particularly in applications requiring joint image-text understanding, reinforcing the benefits of variational modeling in structured representation learning.