The low radiation dose of X-rays is often a dominant source of artifacts in Cone-beam computed tomography (CBCT) images, making it a long-standing and challenging inverse problem. Previous existing data-driven techniques employ 3D decoders with paired huge volumes of training datasets, resulting in limited generalizability and also ignoring the fact of clinical dataset shortage. Even though some implicit neural rendering (INR) methods are focused on per-patient 3D representations under an implicit coordinate to enhance the final reconstructions, they often struggle to simultaneously achieve 3D-consistent and detailed results in the case of extremely sparse views. In this work, we first unify recent advances in INR and probabilistic generation, which propose a geometry-informed score distillation sampling technique for 3D CBCT imaging. In particular, the framework distills robust prior knowledge from the pre-trained 2D axial diffusion models and incorporates plug-and-play geometric information of the measured process to refine the neural radiance field, aiming for high-quality and coherent reconstructions across all dimensions of CBCT volume. We conduct experiments on several challenging in-distribution and out-of-distribution public CT datasets without any retraining. Both quantitative and qualitative assessments demonstrate that our approach outperforms recent works and exhibits superior generalizability.