Integrating deep learning models into medical imaging has greatly improved illness identification, particularly in the setting of COVID-19 diagnosis from chest radiographs. However, the susceptibility of these models to adversarial attacks poses a challenge to their reliability in clinical practice. This study conducts a comprehensive empirical analysis to assess the effectiveness and vulnerabilities of the VGG16 model to adversarial manipulations, focusing on COVID-19 detection. Utilizing Local Interpretable Model-Agnostic Explanations (LIME), we visualize and compare the features contributing to VGG16 model predictions before and after adversarial attacks using the Fast Gradient Sign Method (FGSM). Our findings pave the way for detecting attacks on COVID-19 detection models, enhancing robustness and interpretability in medical imaging applications. By incorporating LIME explanations into the decision-making process, we demonstrate a method for distinguishing between clean and perturbed images, thereby improving the reliability of AI-driven diagnostic tools. This approach underscores the critical need for robust adversarial defense mechanisms in clinical AI applications.