Coffee is a major commodity worldwide that supports the livelihoods of millions of people around the world. Detecting plant diseases earlier helps to prevent large damage, minimize economic losses, and protect the agricultural economies of countries heavily dependent on coffee production. Deep learning is the most effective approach for plant disease detection, however, it still faces critical challenges when having limited labeled datasets and not generalized well in realistic field environments. In this work, we propose a two-stage approach consisting of image background removal using a Segment Anything Model 2 with an external automatic prompting mechanism and a self-supervised colorization for coffee disease classification. The first stage focuses on generating clean images by eliminating background noise, while the second stage enhances the robustness of the classifier through self-supervised learning. This approach not only improves the quality of input data through effective background removal but also, more importantly, enhances the classifier’s robustness and generalization capabilities through self-supervised learning, making it highly effective for real-world applications where accurate disease classification is critical. The experimental results show that the self-supervised colorization model outperforms the baseline, achieving an accuracy of 98.18% and an F1 score of 98.2%.