Optical wireless communication (OWC) has emerged as a promising complementary technology for radio frequency (RF) communication due to its distinctive advantages, especially when integrated with the user-centric (UC) philosophy, resulting in significant enhancements in system performance. However, in scenarios that support user mobility, frequent handovers in UC-OWC networks can degrade the user equipment (UE) experience, thereby necessitating the design of a stable and efficient handover strategy. In this paper, we propose a Q-Learning based dual-end interactive handover strategy (QL-DIHO), which dynamically optimizes network association by alternately adjusting the AP and UE ends. To strike a balance between throughput performance and association continuity, we ingeniously incorporate an additional reward term related to historical association retention into the reward function. Our simulations demonstrate that, compared to the stateless handover strategy that ignores historical association, the proposed QL-DIHO strategy maintains superior association continuity, albeit with an acceptable reduction in throughput performance.