Financial markets are characterized by complex interdependencies and regime shifts. A critical yet underexplored aspect lies in the generation of interpretable factors for various financial tasks, such as risk prediction. These factors should adapt to evolving market conditions while maintaining transparency. We propose a general two-stage framework, which separates factor generation from prediction through a modular pipeline consisting of feature-to-factor transformation and factor-to-forecast integration. The RL-based factor generator combines symbolic optimization with Proximal Policy Optimization (PPO) to produce factors grounded in market microstructure indicators. Our framework maintains explicit mathematical formulations, enabling a deeper understanding of economic causality. Experimental results demonstrate two key advantages: 1) RL-generated factors exhibit superior temporal stability and predictive performance, with a stronger correlation to target variables compared to raw features, and 2) downstream prediction models universally benefit from these RL-enhanced factors, improving overall performance without compromising interpretability.