This work addresses the instability and poor interpretability of computational models in single-cell RNA sequencing (scRNA-seq) analysis. We propose a generalizable framework to evaluate the stability of any model that generates pathway-level scores, applying it to both biologically-constrained variational autoencoders (iVAEs) and alternative graph-based methods like PathSingle. The central contribution is a modular and reproducible workflow, orchestrated with Pixi, Prefect and Ray, that automates the systematic comparison of different models across multiple random seeds. The stability of the learned representations (pathway activities) was assessed using metrics for clustering coherence (Adjusted Mutual Information, AMI) and consistency across runs (hyperbolically weighted Kendall’s Tau, \(w_{\tau }\) ). Our framework revealed that iVAEs informed by biological priors are significantly more stable and produce more meaningful cell groupings than randomly connected counterparts, therefore indicating the importance of being informed by meaningful biological entities. While the PathSingle model demonstrated a marginally superior consistency, the informed VAEs offered a better balance between stability and clustering performance. This work provides a robust methodology for assessing diverse pathway scoring models, promoting the development of more reliable and interpretable tools for single-cell analysis.