ETL is a data integration process that combines data from various sources according to the desired requirements, which is then loaded into a data warehouse. Several no-code/low-code ETL platforms are available for creating and customizing ETL processes. Due to the complexities involved, particularly in the data transformation stage, it is necessary to incorporate tests in the ETL workflow to ensure that target data are accurate, consistent, and adhere to the data quality standards of an organization. In this paper, we propose an extensible ETL testing framework for functional testing of ETL workflows, particularly for no/low-code ETL data pipelines. We also implement a prototype to demonstrate functional testing of data pipelines on the CDAP open-source framework. Our prototype has been validated on four different ETL pipelines, modelling a variety of data sources and properties.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Extensible Test Automation for Functional Testing of Low-Code ETL Workflows

  • Afnan Ahmad,
  • Preetodeep Dev,
  • Arup Kumar Chattopadhyay,
  • Meenakshi D’Souza

摘要

ETL is a data integration process that combines data from various sources according to the desired requirements, which is then loaded into a data warehouse. Several no-code/low-code ETL platforms are available for creating and customizing ETL processes. Due to the complexities involved, particularly in the data transformation stage, it is necessary to incorporate tests in the ETL workflow to ensure that target data are accurate, consistent, and adhere to the data quality standards of an organization. In this paper, we propose an extensible ETL testing framework for functional testing of ETL workflows, particularly for no/low-code ETL data pipelines. We also implement a prototype to demonstrate functional testing of data pipelines on the CDAP open-source framework. Our prototype has been validated on four different ETL pipelines, modelling a variety of data sources and properties.