The Job Shop Scheduling Problem (JSSP) is a well-known challenge in operations research and computer science, widely applied in manufacturing. However, classic formulations often neglect practical tool management aspects such as tool compatibility, changeover time, and slot limitations. This paper introduces an extended formulation, namely JSSP with tool management (JSSP-TM), which explicitly incorporates these tool-related constraints. We model the problem as a Markov Decision Process (MDP) and develop a simulation environment to enable Reinforcement Learning (RL) solutions. Several RL-based methods are explored, including Q-learning with a custom reward function, Stable-Baselines3 algorithms such as Proximal Policy Optimization (PPO), Deep Q-Network (DQN), and Advantage Actor-Critic (A2C), together with traditional heuristic-based solutions. Experimental results show that PPO achieves the best results in terms of makespan and average machine utilization in the test case, demonstrating its potential in solving complex, tool-aware scheduling problems.