Scholarlink

This article introduces a novel Data Quality Assessment Methodology (DQAM) tailored to the challenges of Big Data and Machine Learning (ML), particularly in the context of Federated Learning. DQAM offers a standardised, automated approach to evaluate data quality, emphasising completeness, information entropy, data symmetry, and time-series continuity. Implemented in Python and compatible with Pandas DataFrames, DQAM provides rapid, scalable assessment capabilities crucial for modern ML pipelines. Practical validation of DQAM in agriculture industry use cases demonstrates its effectiveness in identifying and quantifying data quality issues, paving the way for more reliable insights and decision-making. By addressing the lack of standardization and automation in existing approaches, DQAM contributes to enhancing the reliability and accuracy of ML models.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Data Quality Assessment Methodology

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Daniela Delinschi,
  • Rudolf Erdei,
  • Emil Pasca,
  • Oliviu Matei

摘要

This article introduces a novel Data Quality Assessment Methodology (DQAM) tailored to the challenges of Big Data and Machine Learning (ML), particularly in the context of Federated Learning. DQAM offers a standardised, automated approach to evaluate data quality, emphasising completeness, information entropy, data symmetry, and time-series continuity. Implemented in Python and compatible with Pandas DataFrames, DQAM provides rapid, scalable assessment capabilities crucial for modern ML pipelines. Practical validation of DQAM in agriculture industry use cases demonstrates its effectiveness in identifying and quantifying data quality issues, paving the way for more reliable insights and decision-making. By addressing the lack of standardization and automation in existing approaches, DQAM contributes to enhancing the reliability and accuracy of ML models.