Plant disease recognition (PDR) is vital for early detection and control of crop diseases, which affect agricultural productivity. However, factors like lighting, noise, motion blur, and background clutter can degrade model performance in uncontrolled environments. This paper presents a framework to assess the robustness of PDR models under these challenging conditions. The key contributions of this work are as follows: (1) the creation of a real-world dataset, PlantVillage-U, comprising plant images captured under diverse environmental conditions, including noise and blur variations, to mimic real agricultural settings; (2) a comparative evaluation of transformer-based models versus CNN-based models, revealing their respective capabilities and limitations in recognizing plant diseases; and (3) an in-depth analysis of the impact of pre-training on model performance, focusing on the adaptability of pre-trained models to new datasets under uncontrolled conditions. The results show that robust architectures and proper augmentation improve model performance in uncontrolled environments, with significant gains over baseline models. The dataset includes a diverse array of crops such as apple, blueberry, cherry, grape, orange, peach, pepper, potato, raspberry, soy, squash, strawberry and tomato, encompassing both common and complex plant diseases.