Facilitating Information Extraction in Education by Translating Questions from Students, Instructors, and Managers Into Knowledge Graph Queries Through Large Language Models
摘要
Educational organizations increasingly rely on digital technologies, which produce large amounts of data about learners, courses, and learning activities. Despite having the potential to support several educational downstream tasks, this wealth of information often remains inaccessible to non-technical stakeholders due to its complexity and fragmentation. Recent advances in large language models (LLMs) provide a solution to this limitation by enabling access to complex and heterogeneous data through natural language (NL). In this work, we explore how LLMs can empower information extraction from educational knowledge graphs by translating NL questions into graph database queries. To address the lack of resources in this area, we propose a pipeline for generating synthetic natural language questions grounded in realistic educational scenarios, reflecting the needs of students, instructors, and administrators. We then benchmark several state-of-the-art LLMs on the task of translating natural language questions into graph database queries. The results indicate substantial room for improvement: the best-performing model, Mistral-small, achieved a Return Results score of 62%, followed by a finetuned Llama 3 at just under 54%. These findings highlight the current limitations of LLMs in reliably querying KGs via NL and point to the need for more robust, domain-specific solutions. Repository: https://github.com/tail-unica/eduquestions-to-cypher .