A Model-Based Approach for Simple Construction and Efficient Evaluation of Dataframes
摘要
In the dynamic field of data analytics, handling diverse, large-scale datasets poses substantial obstacles, with traditional methods, like data warehousing and manual programming, proving insufficient due to limitations in flexibility and efficiency. Dataframes, facilitated by libraries like Pandas, have gained significant popularity. However, their utility is hampered by limitations in quick construction and modification of queries by non-programming experts, as well as handling large datasets and lacking support for big data without additional libraries. To address these gaps, we introduce a novel model-based approach for dataframe analysis based on the Data Virtual Machine (DVM) framework. Building on this framework, we facilitate the construction and analysis of dataframes queries without requiring programming expertise, lowering the entry barrier for beginners and streamlining the analysis process for experts. The proposed tool facilitates easy formulation and modification of dataframe queries, efficient query evaluation, and demonstrates superior performance compared to traditional Pandas implementation as evidenced by our experimental results.