Scholarlink

Large Language Models (LLMs) are increasingly deployed in sensitive contexts where fairness and inclusivity are critical. Pronoun usage, especially concerning gender-neutral and neopronouns, remains a key challenge for responsible AI. Prior work, such as the MISGENDERED benchmark, revealed significant limitations in earlier LLMs’ handling of inclusive pronouns, but was constrained to outdated models and limited evaluations. In this study, we introduce MISGENDERED+, an extended and updated benchmark for evaluating LLMs’ pronoun fidelity. We benchmark five representative LLMs, GPT-4o, Claude 4, DeepSeek-V3, Qwen Turbo and Qwen2.5, across zero-shot, few-shot, and gender identity inference. Our results show notable improvements compared with the previous studies, especially in binary and gender-neutral pronoun accuracy. However, accuracy on neopronouns and reverse inference tasks remains inconsistent, underscoring persistent gaps in identity-sensitive reasoning. We discussed implications, model-specific observations, and avenues for future inclusive AI research.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Do They Understand Them? An Updated Evaluation on Nonbinary Pronoun Handling in Large Language Models

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Xushuo Tang,
  • Yi Ding,
  • Zhengyi Yang,
  • Yin Chen,
  • Yongrui Gu,
  • Wenke Yang,
  • Mingchen Ju,
  • Xin Cao,
  • Yongfei Liu,
  • Wenjie Zhang

摘要

Large Language Models (LLMs) are increasingly deployed in sensitive contexts where fairness and inclusivity are critical. Pronoun usage, especially concerning gender-neutral and neopronouns, remains a key challenge for responsible AI. Prior work, such as the MISGENDERED benchmark, revealed significant limitations in earlier LLMs’ handling of inclusive pronouns, but was constrained to outdated models and limited evaluations. In this study, we introduce MISGENDERED+, an extended and updated benchmark for evaluating LLMs’ pronoun fidelity. We benchmark five representative LLMs, GPT-4o, Claude 4, DeepSeek-V3, Qwen Turbo and Qwen2.5, across zero-shot, few-shot, and gender identity inference. Our results show notable improvements compared with the previous studies, especially in binary and gender-neutral pronoun accuracy. However, accuracy on neopronouns and reverse inference tasks remains inconsistent, underscoring persistent gaps in identity-sensitive reasoning. We discussed implications, model-specific observations, and avenues for future inclusive AI research.