Custard apple is renowned for its unique taste and nutritional value in the southern part of Asia. The fruit is also renowned for catching diseases. Existing studies have mostly been conducted on detecting custard apples’ ripeness and minuscule amounts on identifying diseases. If early detection of diseases were made possible then the farmers can take precautions and save the rest of the fruits from getting caught up. This study utilizes the existing transfer learning models to classify custard apple disease through image input. The dataset was imbalanced, with the help of augmentation the classes were balanced so that the models train well. Experiments with different TL models lead to ResNet50 outperforming the rest of the models with its 98.84% accuracy. Additionally, XAI was performed to visualize the diagnosed area of the custard apple. Regardless of limited research about custard apples’ diseases this study however outperformed the existing studies in some cases. If properly deployed, our study can contribute to the growth of custard apple cultivation through its disease classification ability.