Assessment of large language models’ performance in simulating students’ self-reports of online self-regulated learning skills
摘要
With the rapid advancement of generative artificial intelligence, the use of Artificial Intelligence (AI) simulated participants in sociological surveys and experiments has become an emerging research direction, paving new avenues for optimizing educational research. However, limited studies have explored the performance of Large Language Models (LLMs) in simulating student participation in educational surveys or experiments. This study examines the performance and differences among three LLMs (gpt-4o, o1-preview, and glm-4) in simulating student participation in online self-regulated learning skills surveys within a blended learning environment under three prompt modes. Mode1 included no role information, Mode2 incorporated student characteristics, and Mode3 further added online learning behavior indicators. Through descriptive analysis, reliability and validity analysis and hypothesis testing of LLM-simulated data under these modes, the study revealed that LLMs possess the capability to simulate middle school students as participants for a survey on online self-regulated learning skills. Among the three LLMs, o1-preview demonstrated the highest overall accuracy, generating survey data closely resembling real student samples when provided with minimal student characteristics. The performance of gpt-4o matched o1-preview when additional online learning behavior indicators were included, while glm-4 underperformed. In terms of prompt modes, Mode3 significantly enhanced the structural validity and consistency of hypothesis testing. It is suggested to select appropriate LLMs and design prompts based on comprehensive analysis of educational research needs and available learning characteristics, and pay attention to the problems of cognitive bias of LLMs and insufficient diversity of simulated data. The findings provide valuable insights into improving the performance of LLM-simulated samples for educational research.