The advent of Large Language Models (LLMs) has significantly propelled the popularity and demand for Question Answering (QA), particularly the Retrieval-Augmented Generation (RAG) approach for a plephora of business needs and applications, most notably for user support and assistance of various kinds. Such industrial NLP systems enable the scaling and optimization of business processes, driving efficiency and innovation. Given the pivotal roles of information retrieval and generation in RAG, the need for swift, continuous evaluation of system performance becomes crucial. We introduce the open-source RURAGE framework \(^{1}\) , designed to assess the quality of QA responses through a combination of straightforward lexical analysis, model-based assessments, and uncertainty metrics. Our empirical findings demonstrate that RURAGE’s ensemble of features achieves comparable outcomes to more resource-intensive evaluations utilizing LLM-as-a-judge, facilitating rapid development in industry settings ( https://github.com/mts-ai/rurage ).