Evaluating Human-Large Language Model Alignment in Group Process
摘要
Exploring the alignment of multi-agent systems with human values during group process is an essential step towards the development of artificial general intelligence. In this work, we present a novel approach to systematically evaluate factors that influence the value orientation of large language models (LLMs) in simulating human group process. Our proposed framework, which requires neither fine-tuning nor pre-training, enables LLMs to simulate debates among personas on a given topic, autonomously assess response confidence, and retrieve external information to enhance low-confidence responses. We conduct comparative experiments using the same framework with human participants. Additionally, we introduce a comprehensive set of evaluation metrics to reveal discrepancies in value orientation alignment between LLM systems and human systems in group process.