A Quantitative Evaluation of Natural Language Retrieval Methods for OpenAPI Specifications
摘要
Effective schema-based data generation fundamentally depends on utilising the correct schema. Consequently, the ability to accurately retrieve the schema most relevant to specific user requirements constitutes a critical area of research. This paper investigates three research questions: 1) What is the most effective mechanism for retrieving structured data? 2) How do different representations of this structured data influence retrieval performance? 3) Can a selective reranking approach improve efficiency without substantially compromising effectiveness? To address these questions, this work introduces NL-to-OpenAPI, a novel dataset pairing natural language queries with corresponding OpenAPI specifications. The queries are specifically designed to evaluate both keyword-based and semantic retrieval approaches. Additionally, a novel selective reranking method is proposed that uses a z-score threshold to estimate retrieval confidence, enabling a trade-off between computational efficiency and retrieval effectiveness.