<p>To address the limitations of existing ECG classification methods, such as unimodal input, insufficient integration of clinical prior knowledge, class imbalance, and high model complexity, this paper proposes a lightweight cross-modal alignment network for arrhythmia classification, referred to as BiSAM-ECGNet. The model adopts a triple-branch architecture, constructing multimodal inputs from recurrence plots, raw signals, and pathological sequences. It employs a lightweight Transformer, multi-scale convolutions, and a 1D-CNN to extract spatiotemporal, morphological, and clinical features from the respective modalities. A cross-modal alignment fusion module is designed, leveraging an improved lightweight Transformer to achieve efficient alignment and deep fusion of multimodal features, thereby enhancing the representation of complex heartbeat patterns. To optimize the training process, the AdaSTM optimizer is proposed, combined with the SMOTE oversampling technique, which effectively improves training stability and mitigates the class imbalance problem. Experiments on the MIT-BIH Arrhythmia Database demonstrate that BiSAM-ECGNet achieves an accuracy (Acc) of 99.71% and a mean F1-score of 99.67% in the five-class task, and a mean F1-score of 99.35% in the more challenging nine-class task, significantly outperforming existing mainstream methods. Ablation studies confirm the effectiveness of the multimodal architecture, while t-SNE visualizations further reveal the model’s superior capability for feature discrimination. While maintaining high accuracy, the model contains only 5.6 million parameters, providing a reliable solution for intelligent ECG diagnosis in resource-constrained environments.</p>