Scholarlink

Introduction <p>Large Language Models (LLM) are AI models that can generate conversational content based on a trained specified source of information (corpus). The aim is to use these corpus-trained LLMs to limit the content offered by LLM, then using prompt engineering to teach using Socratic methods.</p> Methods <p>Two chatbots were created and deployed, powered by OpenAI’s GPT-3.5 model, with a medical-school textbook corpus. The first chatbot generates a brief summary and open-ended question. The second chatbot generates a case vignette from its pre-trained clinical cases, prompting users for a diagnosis. Both chatbots reply to the user’s response, commenting on the accuracy and asks further questions to encourage critical thinking. A randomised controlled trial was conducted on two groups comprising third year medical students. One group used both chatbots, while the other read the medical textbook, both for 10&#xa0;min. A 15-question test, developed from validated past-year examination items and reviewed by General Surgery and Internal Medicine faculty, was administered to both groups before and after the intervention.</p> Results <p>Forty students participated in the study. The average of the group before and after reading the textbook (<i>n</i> = 20) are 3.9 +/- 1.0 and 7.6 +/- 1.5 respectively (<i>p</i> &lt; 0.001). The average of the group before and after using the bot (<i>n</i> = 20) are 3.9 +/- 0.9 and 12.8 +/- 1.6 respectively (<i>p</i> &lt; 0.001). The respective increase in results was 3.7 and 8.9.</p> Discussion <p>Medical students’ learning showed a better performance using a LLM based chatbot compared to self-reading of medical information assessed using a standardised test. More studies are required to determine if LLM-based pedagogical methods are superior to standard education.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A Randomised Controlled Trial to Study the Use of Large Language Models Prompt Engineered with Socratic Methods on the Impact of Medical Students’ Learning

  • 期刊论文

  • 发表日期：

    2026年7月29日

查看全文
  • Cai Ling Yong,
  • Clara Lee Ying Ngoh,
  • Mohammad Shaheryar Furqan,
  • James Wai Kit Lee,
  • Andrew Makmur,
  • Ragunathan Mariappan,
  • Kee Yuan Ngiam

摘要

Introduction

Large Language Models (LLM) are AI models that can generate conversational content based on a trained specified source of information (corpus). The aim is to use these corpus-trained LLMs to limit the content offered by LLM, then using prompt engineering to teach using Socratic methods.

Methods

Two chatbots were created and deployed, powered by OpenAI’s GPT-3.5 model, with a medical-school textbook corpus. The first chatbot generates a brief summary and open-ended question. The second chatbot generates a case vignette from its pre-trained clinical cases, prompting users for a diagnosis. Both chatbots reply to the user’s response, commenting on the accuracy and asks further questions to encourage critical thinking. A randomised controlled trial was conducted on two groups comprising third year medical students. One group used both chatbots, while the other read the medical textbook, both for 10 min. A 15-question test, developed from validated past-year examination items and reviewed by General Surgery and Internal Medicine faculty, was administered to both groups before and after the intervention.

Results

Forty students participated in the study. The average of the group before and after reading the textbook (n = 20) are 3.9 +/- 1.0 and 7.6 +/- 1.5 respectively (p < 0.001). The average of the group before and after using the bot (n = 20) are 3.9 +/- 0.9 and 12.8 +/- 1.6 respectively (p < 0.001). The respective increase in results was 3.7 and 8.9.

Discussion

Medical students’ learning showed a better performance using a LLM based chatbot compared to self-reading of medical information assessed using a standardised test. More studies are required to determine if LLM-based pedagogical methods are superior to standard education.