Machine learning methods have recently made significant breakthroughs in precision agriculture. However, deep learning models need enough training data with expert annotations to train the model efficiently. It is reported that publicly available datasets for plant diseases may suffer from data and capture bias, which may lead to incorrect predictions. To address these concerns, we present a novel deep transfer learning approach to classify tomato diseases using a custom dataset and handling capture bias. We performed ablation study to choose the candidate model for our deep learning experiments. After selecting the best model, we used a Bayesian hyperparameter optimization framework to optimize the model hyperparameters. We employed 10-fold stratified cross-validation with 80% training-validation and 20% test sets, where test set was held out during model training. We further performed experiments with classical machine learning model and deep learning models to evaluate the model predictions. Our model outperformed classical baselines and prior deep CNNs reported for tomato leaves, achieving state-of-the-art accuracy, by achieving 99.40% and 99.47% accuracy of validation and test sets, respectively.