Urban rail transit (URT) systems have become crucial components of sustainable transportation networks in modern cities. These systems consume substantial amounts of energy, contributing significantly to operational costs and environmental impact. To reduce the energy cost, previous works have adopted reinforcement learning (RL) to optimize the controlling parameters that minimizes the energy cost while ensuring performance. However, RL requires exploration-and-exploitation to interact with the environment, which is costly or even dangerous in real URT systems. This paper proposes a novel model-based offline RL approach for optimizing energy efficiency in URT systems. The basic idea is to build a transition model that can be utilized for policy optimization. To improve the generalization, our method leverages physical laws as regularizers and employs Monte Carlo dropout to quantify model uncertainty. The regularizer and uncertainty are further used to penalize the reward function for conservative policy optimization. Experiments demonstrate that our approach achieves average 20% energy consumption reduction compared to conventional methods while maintaining system stability and performance.