Scholarlink

<p>Classifying policy documents into policy issue topics has been a long-time effort in political science and communication disciplines. In this work, we use the GPT 3.5, 4, 4o, Gemini 1.0 pro, 1.5 pro and 1.5 Flash models from OpenAI and Google respectively, which are pre-trained instruction-tuned Large Language Models (LLM), to classify congressional bills and hearings into the Comparative Agendas Project’s 21 major policy topics. We propose three use-case scenarios and estimate overall weighted F1 scores ranging from 0.44 to 0.82 depending on scenario and LLM models employed. The three scenarios aim at minimal, moderate, and major human interference, respectively. Our results point towards the insufficiency of complete reliance on instruction tuned LLMs, an increasing accuracy along with the human effort exerted, and a surprisingly high accuracy achieved in the most humanly demanding use-case. Our superior use-case, which combined GPT 4 and Gemini 1.5 Pro achieved 0.82 weighted F1 score on the 83% of the data in which the two models agreed. Benchmarking against Babel, a custom trained algorithm for this use case, Babel presents with a 13 and 16 percentage point higher accuracy. Future research, practical considerations, and implications of future LLM developments are discussed.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Replacing or enhancing the human coder? Multiclass classification of policy documents with large language models

  • 期刊论文

  • 发表日期：

    2025年2月10日

查看全文
  • Erkan Gunes,
  • Christoffer Koch Florczak

摘要

Classifying policy documents into policy issue topics has been a long-time effort in political science and communication disciplines. In this work, we use the GPT 3.5, 4, 4o, Gemini 1.0 pro, 1.5 pro and 1.5 Flash models from OpenAI and Google respectively, which are pre-trained instruction-tuned Large Language Models (LLM), to classify congressional bills and hearings into the Comparative Agendas Project’s 21 major policy topics. We propose three use-case scenarios and estimate overall weighted F1 scores ranging from 0.44 to 0.82 depending on scenario and LLM models employed. The three scenarios aim at minimal, moderate, and major human interference, respectively. Our results point towards the insufficiency of complete reliance on instruction tuned LLMs, an increasing accuracy along with the human effort exerted, and a surprisingly high accuracy achieved in the most humanly demanding use-case. Our superior use-case, which combined GPT 4 and Gemini 1.5 Pro achieved 0.82 weighted F1 score on the 83% of the data in which the two models agreed. Benchmarking against Babel, a custom trained algorithm for this use case, Babel presents with a 13 and 16 percentage point higher accuracy. Future research, practical considerations, and implications of future LLM developments are discussed.