Extensible Test Automation for Functional Testing of Low-Code ETL Workflows
摘要
ETL is a data integration process that combines data from various sources according to the desired requirements, which is then loaded into a data warehouse. Several no-code/low-code ETL platforms are available for creating and customizing ETL processes. Due to the complexities involved, particularly in the data transformation stage, it is necessary to incorporate tests in the ETL workflow to ensure that target data are accurate, consistent, and adhere to the data quality standards of an organization. In this paper, we propose an extensible ETL testing framework for functional testing of ETL workflows, particularly for no/low-code ETL data pipelines. We also implement a prototype to demonstrate functional testing of data pipelines on the CDAP open-source framework. Our prototype has been validated on four different ETL pipelines, modelling a variety of data sources and properties.