In recent years, Explainable Artificial Intelligence (xAI) has emerged as a key area of interest in deep learning application, with a surge in inquiries, articles, and conferences. The integration of AI into various domains, particularly defense and surveillance, has been driven by advancements in computational photography and AI methods in Computer Vision. However, the reliance on black-box deep learning models without interpretability has led to challenges in finding explanations for their outputs. Modern surveillance systems using object detection and image classification algorithms often misjudge objects, resulting in increased false positives due to external factors. These black-box models lack justification or interpretation, making them unreliable for autonomous operation. To tackle this challenge, xAI techniques have been used for generating explanations/interpretations as to why a certain model classifies or detects certain objects. In this research endeavor, we introduce the xAI approaches utilized in a critical domain, i.e., Maritime Defense and Surveillance with four pre-trained Computer Vision models. Our key observation is that Local Interpretable Modelagnosic Explanations (LIME) provide good result in pinpointing the features contributing the most in the prediction. We intend to address why a model misclassifies certain images/objects (caused by color gradient differences around the area of interest) to make them more robust using qualitative predictions.