Human-Machine Consistency Verification of Text Empathy Automated Scoring in Online Peer Support Scenarios
摘要
This study developed and validated an AI-driven training system designed to enhance empathic accuracy among university peer-support volunteers in online mental-health settings. By fine-tuning the GLM-4 language model on 7 k authentic peer-support dialogues conducted in Chinese universities, we created a high-fidelity simulated client (Patient Bot-GLM-4) that mirrors genuine help-seeking language and emotional states. A dual-encoder architecture, grounded in the EPITOME empathy framework, delivers real-time, explainable feedback on three core empathic dimensions: emotional reaction, interpretation, and exploration. Expert annotations of 5,700 dialogue turn revealed strong human–machine agreement (P < 0.001) and acceptable consistency (QWK ≥ 0.40) across all dimensions, indicating that the system can plausibly assume portions of human supervision. In a subsequent empathic-identification task, the proposed model significantly outperformed RNN, LSTM, and GPT-2 baselines in both accuracy and weighted F1 on each dimension, underscoring the efficacy of task-specific architecture and fine-tuning strategies. The system thus offers a scalable, low-cost, and interpretable solution for training peer helpers and has the potential to augment human oversight within preventive mental-health infrastructures.