Scholarlink

<p>Temporal references to future, past or ongoing events are ubiquitous in natural language. Yet, there is no comprehensive benchmark dataset allowing to assess the ability of Large Language Models (LLMs) to correctly resolve these references. In this work we close this gap and provide such a benchmark that supports the evaluation of the interpretation of <i>Explicit</i> (e.g., “on 30.12.2024”), <i>Implicit</i> (e.g., “yesterday”) and <i>Vague</i> (e.g., “recently”) temporal references. We introduce TRAVELER, a synthetic question-answering benchmark that evaluates a system’s ability to perform temporal reasoning with respect to a set of past events. The benchmark comprises 3,300 English questions, categorized by their temporal reference and automatically generated via four templates from events sets that contain 5-100 events from a household domain. For the <i>Vague</i> category, ground-truth answers were established via human surveys on Prolific, following a procedure inspired by Kenneweg et al. As reported by Kenneweg et al. (in: Proceedings of the Workshop on Cognitive Aspects of the Lexicon @, Torino, 2024). LREC-COLING 2024). To demonstrate the benchmark’s applicability, we evaluate four state-of-the-art LLMs on it. All benchmarked LLMs can answer questions over events sets with a handful of events and <i>Explicit</i> temporal references successfully, but performance clearly deteriorates with larger event set length and when temporal references get less explicit. Notably, the <i>Vague</i> question category exhibits the lowest performance across all models. TRAVELER exposes limitations in current LLMs’ event temporal reasoning capabilities: Their performance clearly deteriorates with longer event sets and when including<i>Vague</i> temporal references. The benchmark, which is publicly available (<a href="https://gitlab.ub.uni-bielefeld.de/s.kenneweg/TRAVELER">https://gitlab.ub.uni-bielefeld.de/s.kenneweg/TRAVELER</a>) also offers the possibility to test the event-temporal reasoning capabilities of other models beyond those tested in this work.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

TRAVELER: A Benchmark for Evaluating Temporal Reasoning Across Vague, Implicit and Explicit References

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年4月22日

查看全文
  • Svenja Kenneweg,
  • Jörg Deigmöller,
  • Philipp Cimiano,
  • Julian Eggert

摘要

Temporal references to future, past or ongoing events are ubiquitous in natural language. Yet, there is no comprehensive benchmark dataset allowing to assess the ability of Large Language Models (LLMs) to correctly resolve these references. In this work we close this gap and provide such a benchmark that supports the evaluation of the interpretation of Explicit (e.g., “on 30.12.2024”), Implicit (e.g., “yesterday”) and Vague (e.g., “recently”) temporal references. We introduce TRAVELER, a synthetic question-answering benchmark that evaluates a system’s ability to perform temporal reasoning with respect to a set of past events. The benchmark comprises 3,300 English questions, categorized by their temporal reference and automatically generated via four templates from events sets that contain 5-100 events from a household domain. For the Vague category, ground-truth answers were established via human surveys on Prolific, following a procedure inspired by Kenneweg et al. As reported by Kenneweg et al. (in: Proceedings of the Workshop on Cognitive Aspects of the Lexicon @, Torino, 2024). LREC-COLING 2024). To demonstrate the benchmark’s applicability, we evaluate four state-of-the-art LLMs on it. All benchmarked LLMs can answer questions over events sets with a handful of events and Explicit temporal references successfully, but performance clearly deteriorates with larger event set length and when temporal references get less explicit. Notably, the Vague question category exhibits the lowest performance across all models. TRAVELER exposes limitations in current LLMs’ event temporal reasoning capabilities: Their performance clearly deteriorates with longer event sets and when includingVague temporal references. The benchmark, which is publicly available (https://gitlab.ub.uni-bielefeld.de/s.kenneweg/TRAVELER) also offers the possibility to test the event-temporal reasoning capabilities of other models beyond those tested in this work.