Scholarlink

Visual explainability in healthcare is often subjective, contextually limited, and inadequate in distinguishing correlation from causality. In this study, we propose a privacy-preserving framework that integrates Federated Learning (FL) with a medical Vision-Language Model (VLM) to generate visual-textual explanations for clinical decision support. The framework consists of a conventional deep learning model trained locally within a federated setting to produce visual explanations while preserving data privacy. These visual outputs are subsequently processed by a fine-tuned, domain-specific VLM to generate clinically meaningful textual interpretations. We evaluate the proposed approach using an attention-based Temporal Convolutional Network (TCN) for electrocardiogram (ECG) analysis and a VLM fine-tuned specifically for ECG data. Experimental results show that the model effectively identifies key diagnostic features and arrhythmia markers while providing clinically relevant insights. The model demonstrates strong semantic alignment (BERTScore: \(0.8343 \pm 0.0073\) ) and robust visual grounding (CLIPScore: \(0.8691 \pm 0.0123\) ), maintaining high performance across diverse diagnostic categories. Qualitative evaluation further reveals that the generated interpretations successfully explain the rationale behind specific cardiac classifications and highlight diagnostically relevant regions, thereby enhancing interpretability and clinical utility. While the model simplifies technical language for accessibility, it occasionally downplays critical diagnostic elements. These findings highlight the potential of combining FL and VLMs for interpretable ECG analysis while also pointing to areas for improving diagnostic fidelity and practical applicability.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Enabling Visual and Textual Explanation in Diagnostics: A Federated Learning Approach with Medical Vision-Language Models

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Sileshi Nibret Zeleke,
  • Mario Bochicchio

摘要

Visual explainability in healthcare is often subjective, contextually limited, and inadequate in distinguishing correlation from causality. In this study, we propose a privacy-preserving framework that integrates Federated Learning (FL) with a medical Vision-Language Model (VLM) to generate visual-textual explanations for clinical decision support. The framework consists of a conventional deep learning model trained locally within a federated setting to produce visual explanations while preserving data privacy. These visual outputs are subsequently processed by a fine-tuned, domain-specific VLM to generate clinically meaningful textual interpretations. We evaluate the proposed approach using an attention-based Temporal Convolutional Network (TCN) for electrocardiogram (ECG) analysis and a VLM fine-tuned specifically for ECG data. Experimental results show that the model effectively identifies key diagnostic features and arrhythmia markers while providing clinically relevant insights. The model demonstrates strong semantic alignment (BERTScore: \(0.8343 \pm 0.0073\) ) and robust visual grounding (CLIPScore: \(0.8691 \pm 0.0123\) ), maintaining high performance across diverse diagnostic categories. Qualitative evaluation further reveals that the generated interpretations successfully explain the rationale behind specific cardiac classifications and highlight diagnostically relevant regions, thereby enhancing interpretability and clinical utility. While the model simplifies technical language for accessibility, it occasionally downplays critical diagnostic elements. These findings highlight the potential of combining FL and VLMs for interpretable ECG analysis while also pointing to areas for improving diagnostic fidelity and practical applicability.