In this paper, we implement a deep learning model for photovoltaic (PV) power forecasting using Global Horizontal Irradiance (GHI) values which are the major determiner of photovoltaic cell power output. We use a multi-layer Long Short-Term Memory (LSTM) model combined with explainable AI (XAI) techniques, aimed at improving the interpretability of predictions across various forecasting horizons. The model utilizes Global Horizontal Irradiance (GHI) data, which undergoes thorough pre-processing, including cleaning and down sampling to ensure data quality and computational efficiency. The LSTM model is designed with multiple layers to capture temporal dependencies and nonlinearities, which are crucial for accurately forecasting PV power under variable environmental conditions. To evaluate model performance, multiple error metrics such as R2, MAE, RMSE, and MAPE are utilized. In addition, a benchmark model is built as a reference to compare against the LSTM-based model, providing a baseline for assessing performance improvements. The use of XAI further enables the interpretation of the LSTM model’s predictions, providing an understanding of feature importance and model behavior. We use the SHAP library to perform XAI analysis by calculating Shapley values. We demonstrate how the SHAP library can be used on 3D LSTM data. Furthermore, the SHAP graphs provide a sense of the importance of each feature’s role in the prediction.