Retinex-based unsupervised low-light enhancement methods have demonstrated notable performance without paired data. However, existing Retinex-based unsupervised methods implicitly relax the constraints of Retinex theory and cannot predict the illumination and reflectance exactly, resulting in unstable outcomes. In order to alleviate this issue, we propose a novel framework with stringent consistent constraints for robust Retinex decomposition. Our work is inspired by the spectral characteristics of the low-light images and primarily utilizes the spectral perturbations to establish the training constraints. Specifically, we first investigate the invariant and equivariant components for low-light enhancement under spectral perturbations. Based on these consistency attributes, we design an illumination invariance constraint and a reflectance equivariance constraint for robust decomposition. Furthermore, motivated by the noise distribution under spectral perturbations, we introduce a cross multi-scale noise regularization technique to tackle the severe noise on the reflectance maps. Extensive experiments conducted on diverse datasets have demonstrated the superior performance over state-of-the-art approaches, highlighting its effectiveness and potential for various applications.