Integrating real-world simulations and decision-making scenarios through case studies has been shown to enhance students’ critical thinking and problem-solving skills. Prior research highlighted a gap in faculty capacity to create or access such materials, an issue particularly acute in developing countries. This research explores the potential of three prominent LLMs, namely ChatGPT-4o, Gemini 2.0, and Perplexity, to generate high-quality case studies for graduate engineering education. In this study, we define a high-quality engineering case study as one that is narratively rich, embeds socio-technical and multidisciplinary perspectives, presents an open-ended practical problem, includes plausible and data rich role-playing scenarios, and supports active learning and reflective thinking. Our evaluation employs a multi-layered methodology combining: (1) a linguistic analysis using standard NLP metrics; (2) automated mixed-method assessments using four LLMs; and (3) mixed-method evaluation by three Subject Matter Experts (SMEs). Our findings revealed that while Perplexity produced the most readable content, it failed along with Gemini to meet the minimum required word count. The SMEs unanimously ranked ChatGPT-4o as superior across all performance dimensions. This study reveals several LLM limitations, such as limited narrative flow, lack of depth, compelling storytelling and academic rigor, failure to meet explicit instructional requirements, and instances of fake or inaccurate citations. These challenges reinforce the necessity for a “human-in-the-loop” approach. Our study offers a balanced and evidence-based perspective on the role of LLMs in augmenting, rather than replacing, human expertise in case study development.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Evaluation of Large Language Models in Simulating Real-World Engineering Scenarios and Decision Making: A Comparative Study of ChatGPT-4o, Gemini, and Perplexity

  • Faouzi Kamoun,
  • Farkhund Iqbal,
  • Heni Abidi,
  • Aymen Ben Brik

摘要

Integrating real-world simulations and decision-making scenarios through case studies has been shown to enhance students’ critical thinking and problem-solving skills. Prior research highlighted a gap in faculty capacity to create or access such materials, an issue particularly acute in developing countries. This research explores the potential of three prominent LLMs, namely ChatGPT-4o, Gemini 2.0, and Perplexity, to generate high-quality case studies for graduate engineering education. In this study, we define a high-quality engineering case study as one that is narratively rich, embeds socio-technical and multidisciplinary perspectives, presents an open-ended practical problem, includes plausible and data rich role-playing scenarios, and supports active learning and reflective thinking. Our evaluation employs a multi-layered methodology combining: (1) a linguistic analysis using standard NLP metrics; (2) automated mixed-method assessments using four LLMs; and (3) mixed-method evaluation by three Subject Matter Experts (SMEs). Our findings revealed that while Perplexity produced the most readable content, it failed along with Gemini to meet the minimum required word count. The SMEs unanimously ranked ChatGPT-4o as superior across all performance dimensions. This study reveals several LLM limitations, such as limited narrative flow, lack of depth, compelling storytelling and academic rigor, failure to meet explicit instructional requirements, and instances of fake or inaccurate citations. These challenges reinforce the necessity for a “human-in-the-loop” approach. Our study offers a balanced and evidence-based perspective on the role of LLMs in augmenting, rather than replacing, human expertise in case study development.