Early detection of skin lesions is crucial for medical diagnosis, as timely identification can significantly improve patient survival rates. However, existing deep learning models often face a trade-off between classification accuracy and computational efficiency, limiting their use in resource-constrained environments. To address this issue, we propose a novel skin lesion classification model, IR-MHAtt, which integrates the IRAtt Block and MHAtt module to improve accuracy. On the one hand, the IRAtt Block improves feature extraction stability by combining an improved inverted residual structure with a self-attention mechanism, which effectively captures multi-scale features while addressing the vanishing and exploding gradient issues that traditional models encounter when processing small-target lesions. On the other hand, the MHAtt module adopts a funnel-shaped design, which cuts computational complexity while preserving global information capture and resolving information loss in existing fusion methods. We evaluate the performance of our IR-MHAtt framework on both HAM10000 and ISIC2019 datasets, achieving classification accuracy of 98.55% on the HAM10000 dataset coupled with exceptional efficiency through a compact architecture containing merely 1.02M parameters and requiring only 0.94G FLOPs. This computationally efficient design significantly outperforms existing methods while maintaining practical clinical applicability for rapid skin lesion diagnosis.