The increasing use of Large language models in recent times has made a drastic impact on the way various Question answering systems are designed and implemented. Different approaches such as model fine-tuning, advanced prompt engineering, few-shot prompting and the incorporation of long context abilities have shown potential for noticeable enhancements in this field. On the other hand, single-handedly relying on Large language models can pose a risk, especially having in mind common drawbacks such as hallucinations, generating factually inconsistent information and the lack of interpretability. With the use of Retrieval Augmented Generation as the base architectural pattern, a part of the previously mentioned concerns can be mitigated. Implementing such systems requires careful consideration of different components and the incorporation of proper mechanisms of evaluation. The need for incorporating these mechanisms especially arises when working with domain-specific systems that are based on the use of low-resource languages, such as the Serbian language. The evaluation proposed in this paper focuses on leveraging the LLM-as-a-judge paradigm for developing a metric driven approach for the improvement of various parts of the proposed Retrieval Augmented Generation pipeline.