Ontology Population Using LLMs: Which Factors Matter?
摘要
While LLMs have proven their performance in many ontology-related tasks, it is not yet known how different factors, such as prompting strategies, ontology structure, and entity label semantics, influence their performance. Our paper aims to address this gap by exploring the effects of a wide range of ontology and LLM characteristics on the accuracy of LLM-driven ontology population. We use three lightweight ontologies and four LLMs to study these effects. The findings suggest that LLMs are capable of performing ontology population with sufficient accuracy but may struggle to infer concept hierarchies, particularly when their embeddings of concept and individual labels are fairly-separated. Few-shot prompting is effective in improving performance of both larger and smaller LLMs. Investigating response variation and consistency, we observe that larger LLMs exhibit less variability over repetitions, with performance reducing as temperature increases. We find that the depth and dispersion of concepts do not influence an LLM’s ability to predict correct hierarchies. In addition, the size of an ontology is not an influential factor for ontology population.