Scholarlink

<p>This methodological paper investigates the viability of using Large Languages Models (LLMs) for the use of qualitative coding of posts to Facebook groups for science teachers. Categories for the coding, and the corresponding prompt for the LLMs, were engineered using an iterative process on progressively larger data sets, whilst checking consistency with human expert coders at each step. Three different versions of OpenAI’s LLMs were used; GPT-3.5 Turbo, GPT-4 Turbo, and GPT-4o, to also investigate the difference in performance between the different versions. The LLMs performed consistently well in qualitative coding, showing moderate to good agreement with human coders. This performance remained strong even when applied to datasets outside the prompt’s original design. Agreement of the LLMs with the human coders was slightly below human–human agreement levels for the two older LLMs, whereas the most recent LLM, GPT4o, achieved an inter-coder reliability that matched that of the human coders. This study proposes several possible strategies for the integration of LLMs into qualitative coding workflows, as well as offering suggestions to improve LLM performance through prompt engineering.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Using large language models to complement humans for the coding of social media interactions between science teachers

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年2月4日

查看全文
  • Robertson Burgess,
  • Katie Waters,
  • Erika Spray,
  • Elena Prieto-Rodriguez

摘要

This methodological paper investigates the viability of using Large Languages Models (LLMs) for the use of qualitative coding of posts to Facebook groups for science teachers. Categories for the coding, and the corresponding prompt for the LLMs, were engineered using an iterative process on progressively larger data sets, whilst checking consistency with human expert coders at each step. Three different versions of OpenAI’s LLMs were used; GPT-3.5 Turbo, GPT-4 Turbo, and GPT-4o, to also investigate the difference in performance between the different versions. The LLMs performed consistently well in qualitative coding, showing moderate to good agreement with human coders. This performance remained strong even when applied to datasets outside the prompt’s original design. Agreement of the LLMs with the human coders was slightly below human–human agreement levels for the two older LLMs, whereas the most recent LLM, GPT4o, achieved an inter-coder reliability that matched that of the human coders. This study proposes several possible strategies for the integration of LLMs into qualitative coding workflows, as well as offering suggestions to improve LLM performance through prompt engineering.