In the dynamic field of data analytics, handling diverse, large-scale datasets poses substantial obstacles, with traditional methods, like data warehousing and manual programming, proving insufficient due to limitations in flexibility and efficiency. Dataframes, facilitated by libraries like Pandas, have gained significant popularity. However, their utility is hampered by limitations in quick construction and modification of queries by non-programming experts, as well as handling large datasets and lacking support for big data without additional libraries. To address these gaps, we introduce a novel model-based approach for dataframe analysis based on the Data Virtual Machine (DVM) framework. Building on this framework, we facilitate the construction and analysis of dataframes queries without requiring programming expertise, lowering the entry barrier for beginners and streamlining the analysis process for experts. The proposed tool facilitates easy formulation and modification of dataframe queries, efficient query evaluation, and demonstrates superior performance compared to traditional Pandas implementation as evidenced by our experimental results.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Model-Based Approach for Simple Construction and Efficient Evaluation of Dataframes

  • Konstantina Zouni,
  • Ioanna Moraiti,
  • Sotirios Angelopoulos,
  • Damianos Chatziantoniou,
  • Verena Kantere

摘要

In the dynamic field of data analytics, handling diverse, large-scale datasets poses substantial obstacles, with traditional methods, like data warehousing and manual programming, proving insufficient due to limitations in flexibility and efficiency. Dataframes, facilitated by libraries like Pandas, have gained significant popularity. However, their utility is hampered by limitations in quick construction and modification of queries by non-programming experts, as well as handling large datasets and lacking support for big data without additional libraries. To address these gaps, we introduce a novel model-based approach for dataframe analysis based on the Data Virtual Machine (DVM) framework. Building on this framework, we facilitate the construction and analysis of dataframes queries without requiring programming expertise, lowering the entry barrier for beginners and streamlining the analysis process for experts. The proposed tool facilitates easy formulation and modification of dataframe queries, efficient query evaluation, and demonstrates superior performance compared to traditional Pandas implementation as evidenced by our experimental results.