This study introduces RL-MTSA, a hybrid framework that integrates reinforcement learning (RL) with multi-thread simulated annealing (MTSA) to enhance multi-objective optimization for vehicle routing problems with time windows and demand priorities (MO-VRPTWDP). Unlike traditional multi-objective metaheuristics that uniformly explore the entire Pareto front, RL-MTSA dynamically adjusts the search direction based on user-defined trade-offs, enabling targeted exploration of preferred regions. The RL agent, trained using Proximal Policy Optimization (PPO) and Advantage Actor-Critic (A2C), guides the optimization process by modifying thread weights in response to each solution’s proximity to the specified trade-off. To evaluate its effectiveness, RL-MTSA is benchmarked against MTSA using a dominance-based comparison across 12 Solomon benchmark instances under five different preference settings. Preliminary results show that RL-MTSA consistently produces solutions that dominate those generated by MTSA, with PPO achieving dominance in over 83% of the cases. These findings demonstrate the framework’s ability to align search behavior with user-specified trade-offs and improve solution relevance in multi-objective routing problems.