Multimodal skin lesion classification methods typically suffer from three limitations: relying on single-scale features, using generic ImageNet pre-training, and treating classification tasks independently without considering medical relationships. We propose MedFuse-GRM, a medically-driven framework with three key components: (1) a multi-scale feature extraction network with cross-attention guided fusion for comprehensive visual representations, (2) contrastive learning pre-training to enhance cross-modal feature consistency between dermoscopy and clinical images, and (3) a medically-guided graph relation module that leverages domain knowledge to model structured connections between diagnostic categories and pathological characteristics. Experiments on the Seven-Point Checklist dataset show MedFuse-GRM achieves 79.7% average accuracy, outperforming state-of-the-art methods by 2.1% points, with ablation studies confirming each component’s substantial contribution.