The importance of Explainable Artificial Intelligence (XAI) in medical research has become increasingly evident, especially with regulations such as the EU AI Act mandating its use. XAI techniques such as LIME and SHAP have been primarily used as tools for interpreting machine learning (ML) models used in disease risk prediction, particularly when analysing tabular Electronic Health Record (EHR) data to identify the top k important features. Although these techniques are widely adopted, their evaluation remains limited. This study uses Clinical Practice Research Datalink (CPRD) data to predict lung cancer risk, highlighting the obstacles posed by highly imbalanced datasets when applying these XAI techniques. By analysing training datasets with varying percentage of lung cancer cases, we assess the consistency of explanations generated by LIME and SHAP across different ML models relative to those trained on a balanced dataset. Our findings reveal that as class imbalance increases, the consistency of feature rankings produced by the same model across different training sets decreases. This finding highlights the practical challenges of implementing XAI techniques in healthcare, particularly in handling data imbalance and the careful application of balancing strategies in risk prediction contexts.