Monkeypox is a rapidly spreading infectious skin disease, presenting significant diagnostic challenges, particularly in resource-limited regions. While traditional diagnostic methods, such as clinical observation and PCR testing, offer high accuracy, they require costly equipment and specialized personnel, limiting their accessibility in under-resourced settings. To address this issue, we propose MonkeyPix, a lightweight Vision Transformer (ViT)-based model designed for fast and accurate monkeypox diagnosis in resource-constrained environments. MonkeyPix enhances standard ViT architecture by introducing pixel-wise self-attention rather than feature-wise processing and incorporating the Shifted Window technique from the Swin Transformer. These modifications reduce the model size by approximately 80% while preserving high diagnostic performance. The model's efficiency allows deployment on low-power devices, making it suitable for real-time use in remote healthcare settings. Furthermore, MonkeyPix demonstrates strong generalizability, showing promise in diagnosing other skin conditions, thus expanding its clinical applicability. By bridging the gap between cutting-edge AI technology and practical public health solutions, MonkeyPix represents a significant advancement in accessible dermatological diagnostics. Future work will explore further optimizations, including multimodal integration and mobile deployment, to enhance its real-world impact.