<p>Model-free RL algorithms perform well in traffic signal control but need many environmental interactions, restricting practical use. model based methods can enhance sample efficiency. However, model errors may impair performance. To mitigate the impact of model errors on the algorithm, we propose a novel hybrid framework that integrates model-based and model-free reinforcement learning. This framework makes full use of past state information and future predicted data, enabling decision-making and forecasting to enhance the accuracy and robustness of traffic signal control. In this framework, we first propose the Lane-Level Traffic Flow Inference Model (LTFIM), which models dynamic adjacency matrices of lane-level traffic flow influenced by intersection traffic signals, predicting the traffic flow state at the next time step. We then present a Model based Multi-Step Value Function Estimation RL method (MMVFE), which performs multi-step predictions using the learned model to estimate value functions and optimize the policy. To stabilize the future reward predictions for the Critic model, We use a historical sliding window mechanism combined with a spatio-temporal Transformer to capture the potential effects of time and space in past states. Experimental results in the SUMO simulation software demonstrate excellent performance in both simulated and real-world scenarios, effectively reducing average vehicle waiting times and improving traffic signal control efficiency.</p>