In the financial field, it is essential to find a set of style factors (also known as risk factors) that accurately model stock volatility. The ideal low-dimensional style factors should balance significance (with high explanatory power) and stability (consistent, no significant fluctuations). However, previous supervised and unsupervised feature extraction methods can hardly address this tradeoff. In this paper, we propose Style Miner, a novel reinforcement learning method for generating style factors. We first formulate the problem as a Constrained Markov Decision Process with explanatory power as the reward and stability as the constraint. Then, we design fine-grained immediate rewards and costs and use a Lagrangian heuristic to balance them adaptively. Experiments on real-world financial data sets show that Style Miner outperforms existing learning-based methods by a large margin and achieves a relatively 10% gain in R-squared explanatory power compared to the industry-renowned factors proposed by human experts.