The escalating global demand for energy, coupled with the depletion of fossil fuel reserves, necessitates a shift toward sustainable and alternative energy sources. In regions like Malaysia, abundant sunlight offers a promising avenue for solar energy adoption. However, the intermittent nature of renewable sources requires effective energy storage solution. This study focuses on optimizing an Energy Management System (EMS) for a Photovoltaic (PV)-integrated battery storage system in a chemical manufacturing plant in Malaysia. Traditional approaches to EMS optimization, such as linear programming, face challenges in handling the dynamic and uncertain nature of renewable energy generation. In response, the reinforcement learning (RL), particularly the n-step Q-learning algorithm, emerges as a viable solution. This machine learning technique enables an agent to make decisions in real-time without relying on forecasted data, crucial for unpredictable variables like load demand and energy generation. This paper investigates the economic benefits of implementing an RL-based EMS in the context of Malaysia’s tariff rate. It also explores how varying n-step values influence the performance and decision-making efficiency of the Q-learning-based EMS. The simulation framework utilizes historical data from a local chemical manufacturing factory, considering constraints of the battery storage system. Results demonstrate that learning the hyper-parameters significantly impact the agent’s performance, highlighting the importance of fine-tuning these hyper-parameters for efficient decision-making. Utilizing a larger n-step value in the algorithm enhances the agent’s decision-making in battery operations, considering cumulative rewards over multiple upcoming time intervals. The EMS, optimized through RL, shows robustness and adaptability, while reducing the cost of energy.