Accurate diagnosis of dermatological conditions remains a critical challenge in medical imaging, primarily due to visual similarities across diseases and limited annotated data. We introduce CAMF-SkinNet, a Cross-Attention Multimodal Fusion framework that integrates visual, textual, and dermatology-specific expert knowledge for robust and clinically relevant diagnosis. CAMF-SkinNet combines three complementary modalities: (i) visual features extracted from skin lesion images using MedSigLIP, (ii) dermatology-specific embeddings from the Google Derm Foundation model, and (iii) context-rich medical captions automatically generated by MedGemma-4B, a large-scale medical vision–language model. Partial fine-tuning of the vision and language backbones, coupled with hierarchical cross-attention modules, enables effective cross-modal alignment and interaction. A lightweight Transformer encoder refines the fused features, and multi-branch classification heads provide auxiliary supervision to preserve modality-specific discriminability. Focal Loss mitigates class imbalance, while early stopping ensures efficient convergence. On a challenging 20-class skin disease dataset, CAMF-SkinNet achieves superior accuracy, macro-F1, and Top-K performance over unimodal and naive fusion baselines, demonstrating the promise of cross-attention-driven multimodal integration for automated dermatological diagnosis.