Skin lesion classification is a core task in automated dermatological analysis. Compared to single modalities, multi-modal data can provide complementary information, accelerating the progress of dermatological classification research. However, multi-modal integration faces challenges of insufficient feature interaction and modality heterogeneity. In this study, we propose a multi-modal fusion framework, CDMFuse, for multi-modal skin lesion classification. CDMFuse driven by two novel modules: the Sum-Difference Feature Interaction Module (SDF) and the Patch-Metadata Cross-Attention Fusion Module (PMCA). The SDF module addresses the insufficient integration of dermoscopic and clinical images in existing methods, which typically fuse features only at late stages, leading to information loss. By introducing sum and difference branches, SDF hierarchically models complementary spatial-channel interactions: the sum branch enhances shared structural features (e.g., lesion boundaries), while the difference branch captures modality-specific details (e.g., microvascular patterns in dermoscopy). The PMCA module tackles the heterogeneity gap between image modalities and metadata, where direct concatenation amplifies redundancy and noise. PMCA employs a cross-attention mechanism to dynamically align image patches with metadata embeddings, resolving structural mismatches. It further integrates content-aware multi-scale convolutions to adaptively adjust receptive fields. The effectiveness of our approach is validated on the publicly available Derm7pt multi-modal skin lesion dataset, achieving an average accuracy of 77.69%, surpassing current state-of-the-art methods and improving the test set accuracy by 1.35%. The method also demonstrates strong performance on XJU-MMSD, further verifying its robustness and superiority in multi-modal skin lesion classification.