Scholarlink

The rise of Large Language Models (LLMs) such as ChatGPT, Microsoft Copilot, and Claude has generated growing interest in their application as educational tools. Recent studies have explored their use not only for content generation but also as instructional partners, particularly in cognitively demanding academic tasks such as literature reviews. However, limited empirical work has assessed their effectiveness as pedagogical agents within structured, dialogic educational settings. This study presents a qualitative, comparative analysis of three advanced LLMs—ChatGPT 5.3, Microsoft Copilot, and Claude 3.5 Haiku—each engaged as a tutor in accordance with Mollick’s AI role taxonomy. Using a 14-step interaction protocol grounded in the ram framework, each model facilitated a dialogically structured reflection in which students collaboratively constructed a SWOT analysis evaluating the model’s own instructional effectiveness. Claude 3.5 Haiku demonstrated the highest levels of protocol compliance, adaptability, and critical depth, consistently providing personalized and reflective feedback. ChatGPT offered accurate and efficient technical support but showed limitations in individualization and procedural fidelity. Microsoft Copilot struggled with dialogic continuity and adherence to the protocol, resulting in weaker overall tutoring performance. These findings underscore the importance of structured prompting and clearly defined pedagogical roles in LLM-based tutoring and highlight the potential of next-generation models like Claude to scaffold high-level academic tasks.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluating Large Language Models as Academic Tutors: A Human-AI Collaboration Approach for Structured Literature Review

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Luca Marconi,
  • Federica Brasca,
  • Paolo Maria Ferri

摘要

The rise of Large Language Models (LLMs) such as ChatGPT, Microsoft Copilot, and Claude has generated growing interest in their application as educational tools. Recent studies have explored their use not only for content generation but also as instructional partners, particularly in cognitively demanding academic tasks such as literature reviews. However, limited empirical work has assessed their effectiveness as pedagogical agents within structured, dialogic educational settings. This study presents a qualitative, comparative analysis of three advanced LLMs—ChatGPT 5.3, Microsoft Copilot, and Claude 3.5 Haiku—each engaged as a tutor in accordance with Mollick’s AI role taxonomy. Using a 14-step interaction protocol grounded in the ram framework, each model facilitated a dialogically structured reflection in which students collaboratively constructed a SWOT analysis evaluating the model’s own instructional effectiveness. Claude 3.5 Haiku demonstrated the highest levels of protocol compliance, adaptability, and critical depth, consistently providing personalized and reflective feedback. ChatGPT offered accurate and efficient technical support but showed limitations in individualization and procedural fidelity. Microsoft Copilot struggled with dialogic continuity and adherence to the protocol, resulting in weaker overall tutoring performance. These findings underscore the importance of structured prompting and clearly defined pedagogical roles in LLM-based tutoring and highlight the potential of next-generation models like Claude to scaffold high-level academic tasks.