Potato leaf blight remains one of the most destructive leaf diseases affecting potato crops in the Andean region, posing a significant threat to food security and the livelihoods of smallholder farmers. This work presents a computer vision framework for the automated detection of early blight (Alternaria solani) and late blight (Phytophthora infestans) in native Andean potato varieties using RGB imagery. Two datasets were employed: a localized dataset (2,766 images) and an extended-localized dataset incorporating additional distractor images that closely resemble those from localized dataset (3,666 images). Three convolutional neural network (CNNs) architectures, a custom CNN, EfficientNetB0, and MobileNetV3, were evaluated for classification performance and interpretability. MobileNetV3 achieved 100% accuracy on the localized dataset and 98.67% on the extended dataset. Grad-CAM visualizations revealed that, under increased variability, MobileNetV3 maintained spatially distributed attention over leaf regions while minimizing reliance on background artifacts, outperforming compared architectures in robustness and interpretability. These results demonstrate that lightweight CNNs trained on localized data augmented with distractor images can effectively mitigate dataset bias and enable the development of efficient, deployable disease detection tools for small-scale agriculture.