Optimization problems characterized by both discrete and continuous variables are ubiquitous across various scientific and engineering disciplines. These mixed-variable challenges present unique difficulties due to their intricate, often discontinuous, solution landscapes and the inherent complexity of navigating such hybrid spaces effectively. To address these multifaceted challenges comprehensively, we introduce a novel hybrid Reinforcement Learning (RL) framework. This innovative approach synergizes the strategic decision-making capabilities of RL, specifically for discrete variable selection, with the precise, efficient search of Bayesian Optimization for continuous variable adjustment. Our framework distinguishes itself through this strategic and dynamic integration of these powerful optimization techniques, enabling it to robustly adapt to the mixed-variable nature of diverse problems. By leveraging RL to intelligently explore discrete decision spaces and simultaneously employing Bayesian Optimization to refine associated continuous parameters, our methodology not only demonstrates remarkable flexibility but also significantly enhances overall optimization performance. Our experiments on synthetic functions and real-world machine learning hyperparameter tuning tasks reveal that our method consistently outperforms traditional RL, random search, and standalone Bayesian optimization in terms of effectiveness and efficiency.