Data lakes offer the flexibility to store large volumes of heterogeneous data with minimal curation. However, this flexibility comes at a cost: traditional keyword-based dataset discovery methods require reliable metadata such as table names or column headers, and become ineffective when this metadata is either missing or incomplete. This issue is especially pronounced in open or poorly maintained data lakes, where the quality of metadata cannot be guaranteed. In this paper, we present CoDD, a system for constraint-based dataset discovery in open data lakes. Instead of querying metadata (query-by-metadata), CoDD profiles datasets by extracting structured facts directly from the data using modular, user-definable components. Users can perform query-by-constraint searches by specifying constraints over the profiled facts in an interactive, question-driven interface. Early results from our user study show that CoDD enables users to find relevant datasets when traditional keyword-based approaches fail due to insufficient or misleading metadata. Furthermore, CoDD performs comparably well even when accurate metadata is available, demonstrating that query-by-constraint is a robust and scalable alternative for dataset discovery in open data lake environments.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

CoDD: A Constraint-Based Dataset Discovery Tool for Open Data Lakes

  • Tim Otto,
  • Christopher Rawald,
  • Stefan Deßloch

摘要

Data lakes offer the flexibility to store large volumes of heterogeneous data with minimal curation. However, this flexibility comes at a cost: traditional keyword-based dataset discovery methods require reliable metadata such as table names or column headers, and become ineffective when this metadata is either missing or incomplete. This issue is especially pronounced in open or poorly maintained data lakes, where the quality of metadata cannot be guaranteed. In this paper, we present CoDD, a system for constraint-based dataset discovery in open data lakes. Instead of querying metadata (query-by-metadata), CoDD profiles datasets by extracting structured facts directly from the data using modular, user-definable components. Users can perform query-by-constraint searches by specifying constraints over the profiled facts in an interactive, question-driven interface. Early results from our user study show that CoDD enables users to find relevant datasets when traditional keyword-based approaches fail due to insufficient or misleading metadata. Furthermore, CoDD performs comparably well even when accurate metadata is available, demonstrating that query-by-constraint is a robust and scalable alternative for dataset discovery in open data lake environments.