Scholarlink

<p>The integration of multiple data modalities through machine learning has significantly enhanced diagnostic accuracy in healthcare. In this paper, we propose a novel fusion-based multimodal framework that leverages chest X-ray images and clinical texts to improve lung disease classification. Our primary contribution is to train a Support Vector Machine (SVM) model that integrates deep learning-based image classification with text-based learning models, creating a multimodal diagnostic system with improved accuracy and efficiency. We further investigate the effectiveness of early and late multimodal fusion strategies, conducting extensive experiments to determine the most effective integration approach. To facilitate this research, we collected a new real-world dataset from Electronic Medical Records at General Hospital of An Giang province, containing both chest X-ray images and clinical texts, a valuable resource for future studies in multimodal healthcare AI. Our experimental results show that the late fusion strategy, which trains a SVM on top of a Vision Transformer (ViT) and Extreme Gradient Boosting trained on Bag-of-Words (XGB-BoW), achieves the highest accuracy of 86.72%. This significantly outperforms unimodal models such as ViT (59.72%) and XGB-BoW (86.15%), as well as the early fusion approach (81.68%). These results show the effectiveness of probability-based late fusion, demonstrating its ability to leverage complementary information from both modalities for enhanced diagnostic accuracy.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Multimodal Approach for Lung Disease Classification: Fusing Chest X-Ray Images and Clinical Texts

  • 期刊论文

  • 发表日期：

    2025年7月2日

查看全文
  • Thi-Diem Truong,
  • Thanh-Nghi Do

摘要

The integration of multiple data modalities through machine learning has significantly enhanced diagnostic accuracy in healthcare. In this paper, we propose a novel fusion-based multimodal framework that leverages chest X-ray images and clinical texts to improve lung disease classification. Our primary contribution is to train a Support Vector Machine (SVM) model that integrates deep learning-based image classification with text-based learning models, creating a multimodal diagnostic system with improved accuracy and efficiency. We further investigate the effectiveness of early and late multimodal fusion strategies, conducting extensive experiments to determine the most effective integration approach. To facilitate this research, we collected a new real-world dataset from Electronic Medical Records at General Hospital of An Giang province, containing both chest X-ray images and clinical texts, a valuable resource for future studies in multimodal healthcare AI. Our experimental results show that the late fusion strategy, which trains a SVM on top of a Vision Transformer (ViT) and Extreme Gradient Boosting trained on Bag-of-Words (XGB-BoW), achieves the highest accuracy of 86.72%. This significantly outperforms unimodal models such as ViT (59.72%) and XGB-BoW (86.15%), as well as the early fusion approach (81.68%). These results show the effectiveness of probability-based late fusion, demonstrating its ability to leverage complementary information from both modalities for enhanced diagnostic accuracy.