Classifying objects into specific categories is a significant yet challenging task in machine learning. The similarities between different species, such as color, petal, sepal and texture, adding to this complexity, making precise classification a demanding problem. Deep learning models, particularly CNNs, have shown conspicuous efficiency in handling such classification tasks. In this study, we evaluate the performance of the transfer learning approach using the EfficientNet (a lightweight and efficient model) and ResNet50 (a comparatively heavier model), both of which are used for fast and accurate feature extraction. Our model has been trained on our own customized dataset, which has six distinct flower categories, and further evaluated using public dataset for comparative analysis. The data preprocessing technique - Context-Aware Adaptive Augmentation(CAAA) was applied to ameliorate feature representation and increase our classification accuracy. To assess the model’s performance, we employed key metrics such as the F1 score, recall, precision, accuracy, and loss analysis. The model achieved an accuracy of 99.11% on our dataset and 93.2% on the external dataset, showcasing its universal applicability. Furthermore, the experimental results underscore the effectiveness of MobileNetV2 in accurately distinguishing visually similar categories, highlighting its potential for real-world classification tasks.