As the deployment of complex black-box machine learning models expands into critical areas like healthcare and criminal justice, the necessity for interpretable explanations of their predictions intensifies. While Local Interpretable Model-agnostic Explanations (LIME) are widely used, they suffer from instability due to random out-of-distribution perturbations and low local fidelity—how closely explanations mirror the model’s behavior locally. These issues arise because small sample weights from perturbed samples prioritize regularization excessively, and perturbed samples are non-local and biased, compromising local fidelity and exposing vulnerabilities to adversarial attacks. In response, we propose KDLIME, an improved method that extends LIME through a KNN-kernel density-based perturbation using in-distribution samples. KDLIME achieves greater local fidelity and resiliency to adversarial challenges by utilizing an unbiased and local empirical distribution. Our extensive evaluation across various real-world datasets demonstrates that KDLIME effectively addresses the issue of producing innocuous explanations that do not adequately capture inherent biases. It also achieves high stability and local fidelity, enhancing the reliability and accuracy of model interpretations.