Reinforcement Learning (RL) techniques, increasingly used in real-world applications, offer a promising alternative for addressing complex challenges in logistics, such as vehicle routing. This study explores how two of the most widely known RL techniques, SARSA and Q-Learning, perform when solving the Capacitated Vehicle Routing Problem (CVRP). We evaluate their performance in terms of solution quality and computational effort across different training strategies, including sequential, parallel, and a mixed alternating approach. Our results show that, while both methods effectively optimize vehicle routing, SARSA outperforms Q-Learning in most scenarios considered. Furthermore, their performance varies depending on the chosen training strategy. In particular, the alternating approach demonstrates a more balanced trade-off between exploration and exploitation, leading to faster convergence and improved route optimization in most cases. These findings not only confirm the potential of RL-based methods to enhance decision-making in logistics but also highlight the impact of key methodological choices, such as the selected RL algorithm and the training and solving strategy, on overall performance.