Training nurses in therapeutic communication is crucial, particularly in mental and behavioral health care. SBIRT (Screening, Brief Intervention, and Referral to Treatment) method equips nurses to identify risky substance use, provide early intervention, and refer individuals to treatment. Traditional SBIRT training often involves subjective and time-intensive evaluations. This paper proposes a scalable, automated multi-agent evaluation system using large language models (LLMs) to simulate patient interactions and analyze nurse-patient conversations, ensuring objective and consistent assessments. By employing dual LLM agents for relevance ranking and reasoning, the system enhances the training process by providing actionable, explainable feedback, advancing both scalability and reliability. We assess our system’s effectiveness by comparing it to a single LLM-agent approach, analyzing improvements in evaluation consistency, feedback quality, and overall training impact. Additionally, we benchmark the performance of various LLMs to determine their suitability for therapeutic communication assessment, highlighting differences in reasoning, relevance ranking, and explainability.