Lesion detection in stroke patients is critical for effective diagnosis and rehabilitation, yet resource-constrained clinical settings demand efficient models. This study introduces MobileFormer, a lightweight CNN-Transformer hybrid architecture, for lesion detection in stroke patients using the ATLAS v2.0 dataset of 1271 T1-weighted MRI scans. MobileFormer integrates an enhanced AdaptiveMobileNet (AMNet) for feature extraction and a novel LiteFormer with Adaptive Token Pooling (ATP) and Lightweight Multi-Head Self-Attention (LMHSA), achieving a balance of performance and efficiency. Our experiments reveal that MobileFormer performs well, with a a mAP@0.5 of 80.8% and an IoU of 76.7%, coming very close to MobileH-Transformer’s results of 81.0% mAP@0.5 and 76.8% IoU. However, MobileFormer uses fewer resources, needing only 1.20 GFLOPs compared to MobileH-Transformer’s 2.00 GFLOPs, and it runs faster at 65 FPS versus 60 FPS. When we looked at the qualitative results, MobileFormer showed strong lesion localization, though it struggled a bit with smaller lesions, much like the other models. Overall, its balance of accuracy and efficiency makes MobileFormer a great option for real-time stroke lesion diagnosis, especially in settings with limited computational resources.