Deep Learning Models (DLM) gave human beings a leap in all the tasks they had to do manually in vision, like image classification, object detection, and image recognition. After the introduction, these models can be attacked with little information or even without any information about the model. It became important to interpret the reason behind it to make it robust because these days machine learning models are applied to many security-critical areas like automotive vehicles and disease prediction. Currently, available evaluations are based mainly on feature analysis. A few papers focus on visual analysis, but those are not based on the recently introduced explainable AI tool, which gives better visual interpretation of image parts, contributing more to making decisions for the target class. In this paper, we evaluate 4 CNN-based deep-learning models on transferable adversarial attacks generated using ResNet50 for image classification. We are generating adversarial attacks by DeepFool and Projected Gradient Decent (PGD) attack reasons behind particular models’ decision-making in classifying adversarial images to classified classes that will help in understanding the robustness of deep models as well as attack and defense mechanisms of existing methods.