Large Language Models (LLMs) have acquired vast amounts of knowledge through extensive pre-training on large corpora. However, the depth of their understanding of ontology knowledge remains unclear. Ontology is about what types of entities exist, how they are grouped into categories, and how they are related to one another. Ontology knowledge provides a standardized approach to knowledge representation that aligns with human cognition. This study aims to analyze the comprehension and mastery of ontological knowledge within LLMs. In this paper, we proposes a comprehensive benchmark that includes generative ontology knowledge data, focusing both on classes and properties. We develop an extensive evaluation framework and design various tasks aiming to evaluate the memorization and utilization of ontological knowledge. The experimental results indicate that, while LLMs can memorize ontological knowledge, they struggle with truly comprehending and utilizing it effectively.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Comprehensive Ontology Knowledge Evaluation System for Large Language Models

  • Xiaotong Qin,
  • Tong Zhou,
  • Yubo Chen,
  • Kang Liu,
  • Jun Zhao

摘要

Large Language Models (LLMs) have acquired vast amounts of knowledge through extensive pre-training on large corpora. However, the depth of their understanding of ontology knowledge remains unclear. Ontology is about what types of entities exist, how they are grouped into categories, and how they are related to one another. Ontology knowledge provides a standardized approach to knowledge representation that aligns with human cognition. This study aims to analyze the comprehension and mastery of ontological knowledge within LLMs. In this paper, we proposes a comprehensive benchmark that includes generative ontology knowledge data, focusing both on classes and properties. We develop an extensive evaluation framework and design various tasks aiming to evaluate the memorization and utilization of ontological knowledge. The experimental results indicate that, while LLMs can memorize ontological knowledge, they struggle with truly comprehending and utilizing it effectively.