Retrieval-Augmented Multi-modal Representation Learning for ECG Analysis
摘要
Accurate ECG diagnosis requires jointly modeling temporal dynamics, waveform morphology, multi-lead spatial correlations, and high-level clinical semantics. To effectively capture all these aspects, a multimodal approach is required. However, the lack of textual annotations in most ECG datasets poses a significant challenge. In addition, existing approaches seldom tackle rare cases and distribution shifts encountered in real-world scenarios. We propose a retrieval-augmented multimodal representation learning framework that integrates raw 12-lead ECG and time-frequency images, as well as a template-based text modality, which is produced by a Text Reinforcement Model when human-written interpretations are unavailable. The sequence branch models the ECG at three granularities (patient, lead, and QRS-guided segments) to preserve physiological cycles and inter-lead interactions. The image branch adopts lead-aware encoding and dynamic view selection to efficiently extract complementary morphological features. To improve robustness to rare patterns and cross-domain shifts, we introduce RAG-S, which retrieves the nearest patient-level prototypes from a tailorable time-series embedding bank and incorporates them into the patient representation via a lightweight gated residual module. Experiments on four public ECG benchmarks show consistent improvements over strong uni- and multimodal baselines in downstream classification.