Saffron is an expensive cash crop obtained from the flowers of Saffron plant. Adulteration is one of the major menaces in the marketing of Saffron which needs to be addressed globally on priority basis. Multiple Saffron adulteration prediction systems have been proposed so far to predict the real and adulterated Saffron samples. However majority of these methods are based upon chemical and non-chemical approaches. The potential disadvantages associated with these approaches are that they are very expensive, complicated and need advanced technology and highly skilled experts. In this research work, a Saffron adulteration prediction system using machine learning has been proposed to predict the given Saffron samples into real or fake classes. The novelty of the system comes from leveraging the power of transfer learning networks to extract deep features from the images for improving the performance of machine learning models. The major advantages of the proposed system is that it is non-destructive, low-cost, and does not require an expert interference. Seven machine learning models have been used for classification including Gradient Boosting, XGB Classifier, AdaBoost, Random Forest, K-Nearest Neighbour (KNN), Support Vector Machine (SVM) and Decision Tree (DT) model. The SVM classifier outperformed all the other classifiers and gave an accuracy of 96.48 \(\%\) . The performance of all the models was also compared with results of existing literature, and it was observed that the proposed feature extraction method and machine learning models performed well.