Scholarlink

Accurate clinical diagnosis requires comprehensive analysis of medical imaging and patient narratives. However, current computer-aided diagnosis methods focus primarily on imaging modalities while neglecting the integration of patient-reported clinical narratives, due to the scarcity of high-quality patient narratives and the limitations in multimodal information fusion. To address these issues, we propose a dual-component framework consisting of: 1) a Retrieval Augmented Patient Narratives Generation Module (RANGM) that employs a retrieval-enhanced mechanism to guide pre-trained large language models in generating clinically plausible patient narratives; and 2) a Multimodal Information Balanced Fusion Network (MIBF-Net) incorporating our novel Information Balanced Fusion Attention (IBFA) module for effective cross-modal integration, along with a Modal Prediction-Divergent Loss (MP-Loss) to enhance the model’s ability to diagnose samples that have ambiguous single modal prediction distribution. Owing to the plug-and-play design, our MIBF-Net can integrate with existing imaging-based state-of-the-art methods. Extensive experiments demonstrate significant performance improvements of 2.3%–4.6% on the HAM10000 dataset and 3.8%–6.4% on the ISIC2019 dataset. Our code is publicly available at https://github.com/sysu19351118/MIBF-Net .

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

MIBF-Net: Multi-Modal Information Balanced Fusion Network for Clinical Diagnosis via Patient Narratives and Lesion Image

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Zixuan Tang,
  • Bai Sun,
  • Shidan He,
  • Yuan Hong,
  • Dongdong Yu,
  • Zhenzhong Liu,
  • Mengtang Li,
  • Bin Chen,
  • Shen Zhao

摘要

Accurate clinical diagnosis requires comprehensive analysis of medical imaging and patient narratives. However, current computer-aided diagnosis methods focus primarily on imaging modalities while neglecting the integration of patient-reported clinical narratives, due to the scarcity of high-quality patient narratives and the limitations in multimodal information fusion. To address these issues, we propose a dual-component framework consisting of: 1) a Retrieval Augmented Patient Narratives Generation Module (RANGM) that employs a retrieval-enhanced mechanism to guide pre-trained large language models in generating clinically plausible patient narratives; and 2) a Multimodal Information Balanced Fusion Network (MIBF-Net) incorporating our novel Information Balanced Fusion Attention (IBFA) module for effective cross-modal integration, along with a Modal Prediction-Divergent Loss (MP-Loss) to enhance the model’s ability to diagnose samples that have ambiguous single modal prediction distribution. Owing to the plug-and-play design, our MIBF-Net can integrate with existing imaging-based state-of-the-art methods. Extensive experiments demonstrate significant performance improvements of 2.3%–4.6% on the HAM10000 dataset and 3.8%–6.4% on the ISIC2019 dataset. Our code is publicly available at https://github.com/sysu19351118/MIBF-Net .