Are LLMs adequate SPARQL query generators? Investigating zero-shot NL-to-SPARQL translation
摘要
The growing importance of Semantic Web technologies and, in particular, Knowledge Graphs (KGs) has increased the need for systems capable of encoding Natural Language (NL) questions into KG queries, such as SPARQL. However, while many template-based Knowledge Graph Question Answering (KGQA) systems have been developed to address this challenge, they often lack flexibility and are primarily designed for general-purpose KGs like Wikidata and DBpedia. At the same time, recent advances in Large Language Models (LLMs) have facilitated the automated construction of custom KGs from unstructured data. As domain-specific KGs become more prevalent, the ability to query them effectively without relying on extensive training data (typically unavailable for these domain-specific KGs) has become a crucial bottleneck. Therefore, in this paper, we explore the potential of LLMs for NL-to-SPARQL translation in a zero-shot scenario. We evaluate 14 instruction-tuned models from diverse families, including general-purpose, coding-specialized, and reasoning-focused architectures, covering a broad range of sizes and capabilities from several model families, such as GPT, Llama, Phi, Cohere, Mistral, Qwen, and DeepSeek. Our evaluation shows that several prompted LLMs achieve high accuracy (> 0.80) on simpler queries, while performance decreases to between 0.30 and 0.70 on more complex contexts. Beyond accuracy, we analyze query categorization, syntax correctness, response consistency, and failure cases, establishing a strong baseline for zero-shot NL-to-SPARQL translation and highlighting clear opportunities for improvement via few-shot learning or fine-tuning.