Leaf disease recognition through image processing has significant applications such as early detection, targeted treatments, and better agricultural practices. Some of the intrinsic challenges in identifying the diseases from the leaves are due to their irregular shapes, different sizes, rich colors, fuzzy boundaries, and cluttered backgrounds. A significant drop in performance under real field conditions has been the trend even though good accuracies have resulted with lab images. The need for efficient and accurate automatic background subtraction methods for leaf images has become evident through logical inference. The proposed approach aims to address these challenges and results in more robust and accurate plant disease recognition in real-world agricultural scenarios. This paper proposes a two-step process for the segmentation and classification tasks of the diseased leaf. A modified and improved version of U-Net is constructed for segmentation by introducing attention gates and residual blocks in the architecture on a representative dataset. The residual blocks help tackle the gradient disappearance and explosion problems, while attention helps the network understand which parts of the spatial representation of the image are necessary for the classification task at hand. Various convolutional network variations were trained for classification, and the best one was chosen for the model. Experimental results on a dataset of diseased plant leaf images demonstrate that the proposed method significantly enhances the accuracy and efficiency of segmenting diseased leaf images.