The development of the Internet of Vehicles (IoV) has brought significant benefits to urban transportation and people’s lives, with its interconnected nature enabling the realization of various intelligent services. However, in current IoV systems, data is primarily collected by individual vehicles during their operations and transmitted to servers for processing, which undoubtedly poses a risk of data privacy breaches. Fortunately, the emergence of federated learning offers a novel solution to this issue. Nevertheless, there remain three critical issues that need to be addressed: communication restrictions, incentive mechanisms, and unknown information. In this paper, we propose a two-stage client selection algorithm that integrates incentive mechanisms and reinforcement learning, aiming to balance these factors. Through a two-stage selection process, the algorithm significantly reduces the communication pressure on servers. Reinforcement learning is employed to make optimal selection decisions under varying environments with unknown vehicle-end information. Furthermore, the incentive mechanism, not only a necessity in the real world, also optimizes the action space of reinforcement learning. Notably, our proposed method uniquely emphasizes enhancing the utilization of global data, striking a balance between various factors. Relevant experiments have demonstrated the effectiveness of our approach.