Swin-SAINT: A Hybrid Transformer Framework for Multi-modal Skin Cancer Detection Using Structured Metadata and Dermoscopic Images
摘要
Early detection of melanoma and other skin cancers is critical for improving patient outcomes, yet remains challenging due to variability in lesion appearance and limited annotated data. This paper presents Swin-SAINT, a novel hybrid transformer-based framework that integrates structured clinical metadata and unstructured dermoscopic images for robust and interpretable skin cancer classification. The proposed architecture utilizes the Self-Attention and Intersample Attention Transformer (SAINT) for encoding tabular clinical data, and the Swin Transformer for hierarchical visual feature extraction from dermoscopic images. To enhance representation learning under weak supervision, a Deep Embedded Clustering (DEC) module is employed for unsupervised refinement of latent embeddings. A novel Cross-Modality Attention Fusion (CMAF) mechanism is introduced to align and jointly model the complementary information across modalities. Comprehensive experiments on the ISIC 2018 dataset demonstrate that the proposed method outperforms state-of-the-art baselines in accuracy, sensitivity, and AUC. The framework also provides interpretable insights via attention visualization, making it a practical tool for AI-assisted dermatological diagnosis. These results highlight the potential of Swin-SAINT to serve as a scalable, accurate, and trustworthy solution in clinical skin cancer screening.