HERTy-Wiki: A Benchmark for Hierarchical Entity Reasoning and Typing in Wikidata
摘要
Entity typing is central to knowledge engineering, and as large language models (LLMs) are increasingly used to support knowledge graph construction, it becomes essential to assess how reliably they can perform this task. We present HERTy-Wiki (Hierarchical Entity Reasoning and Typing in Wikidata) - a human-verified benchmark for evaluating hierarchical reasoning in LLMs for knowledge graph entity typing (KGET). Unlike existing probing benchmarks that test factual recall, HERTy-Wiki evaluates whether models can select the most specific valid type from limited contextual evidence, reflecting realistic Wikidata scenarios, where editors often work with incomplete or noisy information. HERTy-Wiki comprises 8,767 multiple-choice questions derived from 3,776 Wikidata entities across five domains, with an optional multimodal extension. Each question requires models to discriminate between semantically related types within the same subclass hierarchy. We evaluate several state-of-the-art, reasoning-oriented LLMs under zero-shot, few-shot, and chain-of-thought prompting settings and compare them against a no-context baseline to quantify model reliance on memorised priors. Across models, macro-F1 scores remain around 0.50–0.60, with only modest improvements over the no-context baseline. Our analysis indicates that the emerging hierarchical reasoning abilities of models are overshadowed by strong dependence on memorised priors. These findings highlight fundamental challenges in using LLMs for reasoning in knowledge engineering pipelines, particularly in rapidly evolving or specialised domains with limited pre-training coverage. HERTy-Wiki provides a human-verified benchmark for developing and evaluating genuine reasoning-driven approaches to KGET.