Effective schema-based data generation fundamentally depends on utilising the correct schema. Consequently, the ability to accurately retrieve the schema most relevant to specific user requirements constitutes a critical area of research. This paper investigates three research questions: 1) What is the most effective mechanism for retrieving structured data? 2) How do different representations of this structured data influence retrieval performance? 3) Can a selective reranking approach improve efficiency without substantially compromising effectiveness? To address these questions, this work introduces NL-to-OpenAPI, a novel dataset pairing natural language queries with corresponding OpenAPI specifications. The queries are specifically designed to evaluate both keyword-based and semantic retrieval approaches. Additionally, a novel selective reranking method is proposed that uses a z-score threshold to estimate retrieval confidence, enabling a trade-off between computational efficiency and retrieval effectiveness.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Quantitative Evaluation of Natural Language Retrieval Methods for OpenAPI Specifications

  • Shankar Mathew Palamootil,
  • Jacomine Grobler

摘要

Effective schema-based data generation fundamentally depends on utilising the correct schema. Consequently, the ability to accurately retrieve the schema most relevant to specific user requirements constitutes a critical area of research. This paper investigates three research questions: 1) What is the most effective mechanism for retrieving structured data? 2) How do different representations of this structured data influence retrieval performance? 3) Can a selective reranking approach improve efficiency without substantially compromising effectiveness? To address these questions, this work introduces NL-to-OpenAPI, a novel dataset pairing natural language queries with corresponding OpenAPI specifications. The queries are specifically designed to evaluate both keyword-based and semantic retrieval approaches. Additionally, a novel selective reranking method is proposed that uses a z-score threshold to estimate retrieval confidence, enabling a trade-off between computational efficiency and retrieval effectiveness.