Medical Vision-Language Models (Med-VLMs) excel in clinical image understanding but suffer from hallucinations, producing plausible yet incorrect outputs that risk clinical safety. We present MedDual, a practical dual-decoding framework integrating our Modality-Aware Contrastive Decoding (MACD) with Dynamic Correction Decoding (DeCo). MACD employs tailored perturbations for specific modalities (X-ray, CT, MRI, pathology) to preserve diagnostic features and disrupt spurious correlations, surpassing conventional uniform approaches. DeCo dynamically corrects logits via intermediate layer contrasts for improved factuality. This synergy refines predictions without retraining or external resources. Evaluations on VQA-RAD, SLAKE, and PathVQA show gains up to 6.3% in open-ended recall and 0.9% in close-ended accuracy over baselines like VCD and DoLa. Ablations confirm complementary benefits: MACD enhances visual grounding, DeCo boosts textual accuracy. Our open-source solution advances Med-VLM reliability for healthcare deployment.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

MedDual: A Practical Dual-Decoding Framework for Mitigating Hallucinations in Medical Vision-Language Models

  • Zhe Zhang,
  • Daisong Gan,
  • Zhaochi Wen,
  • Dong Liang

摘要

Medical Vision-Language Models (Med-VLMs) excel in clinical image understanding but suffer from hallucinations, producing plausible yet incorrect outputs that risk clinical safety. We present MedDual, a practical dual-decoding framework integrating our Modality-Aware Contrastive Decoding (MACD) with Dynamic Correction Decoding (DeCo). MACD employs tailored perturbations for specific modalities (X-ray, CT, MRI, pathology) to preserve diagnostic features and disrupt spurious correlations, surpassing conventional uniform approaches. DeCo dynamically corrects logits via intermediate layer contrasts for improved factuality. This synergy refines predictions without retraining or external resources. Evaluations on VQA-RAD, SLAKE, and PathVQA show gains up to 6.3% in open-ended recall and 0.9% in close-ended accuracy over baselines like VCD and DoLa. Ablations confirm complementary benefits: MACD enhances visual grounding, DeCo boosts textual accuracy. Our open-source solution advances Med-VLM reliability for healthcare deployment.