Large Language Models (LLMs) are widely used across various applications, yet their performance varies significantly depending on the task. Some models excel in reasoning, while others perform better in language understanding or factual accuracy, making it challenging to determine the most suitable LLM for specific use cases. To address this, we introduce SallyEval, an evaluation platform designed to identify the optimal LLM for integration into the Sally chatbot, an AI system that assists migrants in Greece. SallyEval employs A/B testing with human preference-based scoring, incorporating Elo rating and adaptive sampling to efficiently rank models based on real-world user interactions. In evaluations, the top-performing model, GPT4o-Latest, achieved an Elo score of 1050, with a 64% win rate. Unlike traditional pairwise comparisons that solely determine a winning response, SallyEval enhances the evaluation process by allowing evaluators to provide detailed feedback through a rating scale that covers multiple qualitative assessment criteria, such as accuracy, contextual relevance, clarity, and conciseness, while also supporting custom benchmarking for domain-specific evaluations.