Scholarlink

With the increasing use of Large Language Models (LLMs) for coding in qualitative research, this paper examines the ways that GPT-4o can code discourse data from a Critical Machine Learning (CML) curriculum for Black middle school girls. We question whether large data representing non-mainstream discourse can be coded successfully by the model, specifically by comparing human-coded data with outputs from three GPT-4o models: unsegmented (base), activity-wise segmentation (stanza), and sub-activity segmentation (sub-stanza). The principles of our study are based in Quantitative Ethnography, and we evaluate the LLM’s performance using Cohen’s Kappa, alongside qualitative justifications generated for each coded line. Even though GPT-4o received detailed definitions of constructs and background of the original curriculum, it showed significantly low agreement across all codes with human coders. Our findings shed light on the lack of transparency by OpenAI on training data, and the danger of semantic flattening on particular, culture-specific discourse in qualitative research. We argue for reflexivity from researchers during and after coding, care with prompt engineering, and the need for more culturally responsive AI tools in qualitative research.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Of Humans and Machines: Evaluating the Efficacy of GPT-4 in Coding Discourse Data

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Omer Zahid,
  • Jen Hsiang-Pan,
  • Golnaz Arastoopour Irgens,
  • Atefeh Behboudi,
  • Alicia C. Lane

摘要

With the increasing use of Large Language Models (LLMs) for coding in qualitative research, this paper examines the ways that GPT-4o can code discourse data from a Critical Machine Learning (CML) curriculum for Black middle school girls. We question whether large data representing non-mainstream discourse can be coded successfully by the model, specifically by comparing human-coded data with outputs from three GPT-4o models: unsegmented (base), activity-wise segmentation (stanza), and sub-activity segmentation (sub-stanza). The principles of our study are based in Quantitative Ethnography, and we evaluate the LLM’s performance using Cohen’s Kappa, alongside qualitative justifications generated for each coded line. Even though GPT-4o received detailed definitions of constructs and background of the original curriculum, it showed significantly low agreement across all codes with human coders. Our findings shed light on the lack of transparency by OpenAI on training data, and the danger of semantic flattening on particular, culture-specific discourse in qualitative research. We argue for reflexivity from researchers during and after coding, care with prompt engineering, and the need for more culturally responsive AI tools in qualitative research.