Understanding implicit inferences in human communication is essential for Large Language Models (LLMs). However, existing evaluation benchmarks are predominantly English-centric and rarely systematically assess the semantic–pragmatic interface. To address these gaps, we introduce EPIc, the first Chinese dataset aimed at entailment, presupposition and scalar implicature across affirmative, negative and conditional contexts. The dataset includes 480 items derived from both natural corpora and semi-automated generation. We evaluate six LLMs using a multiple-choice task. Our results reveal a significant performance disparity, suggesting that while most models achieve near-human accuracy on presupposition, they struggle profoundly with entailment and scalar implicature, especially in negative and conditional contexts. This highlights a critical failure in reasoning about sentence monotonicity and activating scalar alternatives. Our work not only provides the first systematic evaluation of these phenomena in Chinese but also exposes fundamental limitations in the semantic–pragmatic capabilities of current LLMs, offering a valuable resource for future research. Code and dataset are available at https://github.com/6050ZSQ/EPIc .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Do Large Language Models Grasp Implicit Inference?

  • Shiqi Zhu,
  • Xingcheng Li

摘要

Understanding implicit inferences in human communication is essential for Large Language Models (LLMs). However, existing evaluation benchmarks are predominantly English-centric and rarely systematically assess the semantic–pragmatic interface. To address these gaps, we introduce EPIc, the first Chinese dataset aimed at entailment, presupposition and scalar implicature across affirmative, negative and conditional contexts. The dataset includes 480 items derived from both natural corpora and semi-automated generation. We evaluate six LLMs using a multiple-choice task. Our results reveal a significant performance disparity, suggesting that while most models achieve near-human accuracy on presupposition, they struggle profoundly with entailment and scalar implicature, especially in negative and conditional contexts. This highlights a critical failure in reasoning about sentence monotonicity and activating scalar alternatives. Our work not only provides the first systematic evaluation of these phenomena in Chinese but also exposes fundamental limitations in the semantic–pragmatic capabilities of current LLMs, offering a valuable resource for future research. Code and dataset are available at https://github.com/6050ZSQ/EPIc .