With the rapid advancement of autonomous vehicle technology, the development of efficient and stable trajectory tracking algorithms is crucial for achieving autonomous navigation. This paper addresses the issue of insufficient stability in traditional MPC for vehicle trajectory tracking and proposes a Q-learning-based multi-step model predictive control (QM-MPC) algorithm. This algorithm integrates the principles of model predictive control and reinforcement learning to accurately predict the future motion trajectory of autonomous vehicles using MPC. Furthermore, it utilizes Q-learning to perform multi-step deep predictions on the MPC-predicted motion trajectory results and propagates these deep predictions back to the current state. Through this approach, the objective cost function of MPC for autonomous vehicles is optimized, thereby further enhancing the efficiency and stability of trajectory tracking. Experimental results demonstrate that the proposed QM-MPC algorithm exhibits robust stability of Unmanned Ground Vehicle (UGV) in trajectory tracking. Specifically, it significantly enhances stability by 4.303%, 61.965%, 51.481% and 39.024% in path deviation, convergence time, yaw angle deviation and yaw rate, respectively. These findings underscore its considerable utility in autonomous vehicle navigation.