Autonomous navigation plays a crucial role in enabling mobile robots to operate effectively in complex environments. This paper focuses on enhancing navigation performance using Q-learning algorithm, by investigating its implementation with and without epsilon-greedy exploration strategies. The research evaluates performance in different maze sizes, with a specific focus on the average time the agent takes to reach the goal under each condition. Additionally, the study explores the effects of hyperparameter tuning on learning efficiency, convergence speed, and overall success rates. A key feature of the proposed approach is its ability to dynamically adjust alpha and gamma values during training, allowing the agent to autonomously identify optimal parameters. This adaptability enables the algorithm to perform effectively across a wide range of scenarios, rather than being limited to specific cases. Simulations conducted in MATLAB environments highlight the adaptability of Q-learning to resource-constrained systems while maintaining robust decision-making capabilities. By analyzing key performance metrics, this work provides practical insights into optimizing autonomous navigation systems and highlights the versatility of reinforcement learning. The results demonstrate how tailored exploration strategies and parameter adjustments can significantly improve navigation in both simple and complex environments.