Traditionally, text classification research has predominantly focused on extracting single text features, with limited exploration of integrating other modal information (such as speech and images) to enhance classification performance. To address this research gap, we propose the Multimodal Representation for Text Classification (MRTC) framework. This framework aims to boost text classification performance by incorporating speech, image, and text features. Specifically, we employ advanced text-to-speech models to convert text content into audio features. Simultaneously, we retrieve images closely associated with the text content and extract their visual features to further enrich the information dimension of text representation. Subsequently, we utilize an efficient triplet structure network to fuse the speech, image, and text features, thereby constructing a multimodal feature representation for application in text classification tasks. The proposed MRTC framework achieves high-precision text classification across multiple datasets without requiring additional multimodal annotated data. This characteristic not only reduces the cost of data annotation but also enhances the model’s practical flexibility and scalability. To validate the effectiveness of the MRTC framework, we conduct experiments on six distinct text classification tasks. The experimental results demonstrate the significant effectiveness of our MRTC framework across various text classification tasks.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Improving Text Classification Performance Through Multimodal Representation

  • Yujia Wu,
  • Xuan Zhang,
  • Hong Ren

摘要

Traditionally, text classification research has predominantly focused on extracting single text features, with limited exploration of integrating other modal information (such as speech and images) to enhance classification performance. To address this research gap, we propose the Multimodal Representation for Text Classification (MRTC) framework. This framework aims to boost text classification performance by incorporating speech, image, and text features. Specifically, we employ advanced text-to-speech models to convert text content into audio features. Simultaneously, we retrieve images closely associated with the text content and extract their visual features to further enrich the information dimension of text representation. Subsequently, we utilize an efficient triplet structure network to fuse the speech, image, and text features, thereby constructing a multimodal feature representation for application in text classification tasks. The proposed MRTC framework achieves high-precision text classification across multiple datasets without requiring additional multimodal annotated data. This characteristic not only reduces the cost of data annotation but also enhances the model’s practical flexibility and scalability. To validate the effectiveness of the MRTC framework, we conduct experiments on six distinct text classification tasks. The experimental results demonstrate the significant effectiveness of our MRTC framework across various text classification tasks.