Reinforcement Learning (RL) has shown promise in optimizing HVAC control for improved efficiency and cost savings. However, its “black box” nature presents challenges in real-world deployment, where transparency and reliability are critical. This study applies Explainable AI (XAI), specifically Shapley Additive Explanations (SHAP), to interpret RL decision-making and assess its alignment with ex-pert expectations. A methodology is introduced to quantify agreement between RL model explanations and expert-defined control logic using two metrics: Spearman’s Rank Correlation and the Directional Agreement Index (DAI). These metrics evaluate whether the RL model assigns feature importance and directionality in a manner consistent with expert knowledge. The study employs the BESTEST Air test case from the BOPTEST framework, using a four-pipe fan coil unit with an RL controller. Initial results reveal a misalignment between expert expectations and RL-derived feature importance, particularly the RL controller’s reliance on periodic features like solar altitude. Adjustments to feature selection and reward function weights improved agreement scores and led to better energy efficiency and comfort performance. Findings highlight the potential of using XAI to refine RL models, enhancing interpretability and trust in AI-driven control systems. The iterative approach of hypothesis formation, validation, and refinement provides a framework for integrating expert knowledge into RL-based HVAC control, ensuring robust, explainable, and practical AI deployment in building management systems.