Skin cancer detection, crucial for early intervention, faces challenges like subtle visual differences between benign and malignant lesions, the need for expert dermatological knowledge, and the variability in image quality and data availability, hindering automated and accurate diagnosis. Traditional CNN-based models encounter multiple challenges when used for skin cancer detection, but the central issue is high-quality annotated datasets. These datasets also exhibit class imbalance. This paper presents an EfficientViT-L1 model integrated with the CAGA (Cascaded Atrous Group Attention) module for accurately classifying skin lesions, including dangerous types like malignant and benign. The CAGA mechanism has reduced redundancy in Multi-Head Self-Attention to enhance feature extraction while maintaining computational efficiency. The proposed approach detects skin lesions with an accuracy of 90.93% on HAM10000, comprising 10015 images with seven different types of lesions. The work represents a strong, early detection of skin cancer while addressing challenges like class imbalance and precise feature differentiation by classifying the lesion type. Increasing the interpretability of the model can also build a good trust of AI in healthcare.