Accurate ECG diagnosis requires jointly modeling temporal dynamics, waveform morphology, multi-lead spatial correlations, and high-level clinical semantics. To effectively capture all these aspects, a multimodal approach is required. However, the lack of textual annotations in most ECG datasets poses a significant challenge. In addition, existing approaches seldom tackle rare cases and distribution shifts encountered in real-world scenarios. We propose a retrieval-augmented multimodal representation learning framework that integrates raw 12-lead ECG and time-frequency images, as well as a template-based text modality, which is produced by a Text Reinforcement Model when human-written interpretations are unavailable. The sequence branch models the ECG at three granularities (patient, lead, and QRS-guided segments) to preserve physiological cycles and inter-lead interactions. The image branch adopts lead-aware encoding and dynamic view selection to efficiently extract complementary morphological features. To improve robustness to rare patterns and cross-domain shifts, we introduce RAG-S, which retrieves the nearest patient-level prototypes from a tailorable time-series embedding bank and incorporates them into the patient representation via a lightweight gated residual module. Experiments on four public ECG benchmarks show consistent improvements over strong uni- and multimodal baselines in downstream classification.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Retrieval-Augmented Multi-modal Representation Learning for ECG Analysis

  • Shuang Wang,
  • Zhenlong Pang,
  • Xinzhu Xu,
  • Shirui Wang,
  • Li Zhang

摘要

Accurate ECG diagnosis requires jointly modeling temporal dynamics, waveform morphology, multi-lead spatial correlations, and high-level clinical semantics. To effectively capture all these aspects, a multimodal approach is required. However, the lack of textual annotations in most ECG datasets poses a significant challenge. In addition, existing approaches seldom tackle rare cases and distribution shifts encountered in real-world scenarios. We propose a retrieval-augmented multimodal representation learning framework that integrates raw 12-lead ECG and time-frequency images, as well as a template-based text modality, which is produced by a Text Reinforcement Model when human-written interpretations are unavailable. The sequence branch models the ECG at three granularities (patient, lead, and QRS-guided segments) to preserve physiological cycles and inter-lead interactions. The image branch adopts lead-aware encoding and dynamic view selection to efficiently extract complementary morphological features. To improve robustness to rare patterns and cross-domain shifts, we introduce RAG-S, which retrieves the nearest patient-level prototypes from a tailorable time-series embedding bank and incorporates them into the patient representation via a lightweight gated residual module. Experiments on four public ECG benchmarks show consistent improvements over strong uni- and multimodal baselines in downstream classification.