The paper reports on the initial research work investigating the feasibility, trustworthiness and transparency of the decisions driven by the complex Machine Learning models, the aspects especially important in context of mission-critical and high-impact automated control systems, such as the District Heating System. Specifically, the paper demonstrates and discusses the use of feature importances and counterfactual explanations of the local predictions made by the trained heat demand forecasting model. The analysis of the local feature importances by using SHAP method has uncovered humanly interpreted insights into the local model behavior and has confirmed the intuitive knowledge and common practices by the plant operators. Counterfactuals have been generated by using DiCE method. The paper specifically addressed the feasibility of the counterfactuals and proposed Euclidean distances as the simplistic but effective method for selecting the most actionable alternate strategy for achieving the operational goals as indicated in the analysis.