Real-time data processing has become important in many fields such as healthcare, finance, and smart systems. Large volumes of data flow continuously through pipelines, where errors and inconsistencies often affect the final results. Data cleaning and pipeline debugging are needed to maintain data quality, but manual checking is slow and not suitable for real-time systems. This is where Explainable Artificial Intelligence (XAI) agents can help. XAI agents can point out the exact step in the pipeline where something went wrong. These agents not only clean the data but also give clear explanations of what went wrong and what changes were made. This helps data engineers and analysts fix issues quickly without going through every step manually. Many existing tools clean data in the background but do not explain what they do. This makes it hard for users to trust or improve the process. The proposed approach uses real-time XAI agents that detect data errors, fix them, and explain their actions using simple rules and examples. These agents also track the entire pipeline to find issues in logic or flow. Each action taken by the agent is logged with a reason so users can trace the changes. The system was tested on large real-time datasets from public and industrial sources. Evaluation metrics such as accuracy, error detection rate (EDR), and debugging time (DT) showed better performance when compared with traditional data cleaning tools. Users reported higher confidence in the results due to the clear explanations provided by the agents.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Explainable Artificial Intelligence Agents for Data Cleaning and Pipeline Debugging in Realtime

  • Suman Ankampally

摘要

Real-time data processing has become important in many fields such as healthcare, finance, and smart systems. Large volumes of data flow continuously through pipelines, where errors and inconsistencies often affect the final results. Data cleaning and pipeline debugging are needed to maintain data quality, but manual checking is slow and not suitable for real-time systems. This is where Explainable Artificial Intelligence (XAI) agents can help. XAI agents can point out the exact step in the pipeline where something went wrong. These agents not only clean the data but also give clear explanations of what went wrong and what changes were made. This helps data engineers and analysts fix issues quickly without going through every step manually. Many existing tools clean data in the background but do not explain what they do. This makes it hard for users to trust or improve the process. The proposed approach uses real-time XAI agents that detect data errors, fix them, and explain their actions using simple rules and examples. These agents also track the entire pipeline to find issues in logic or flow. Each action taken by the agent is logged with a reason so users can trace the changes. The system was tested on large real-time datasets from public and industrial sources. Evaluation metrics such as accuracy, error detection rate (EDR), and debugging time (DT) showed better performance when compared with traditional data cleaning tools. Users reported higher confidence in the results due to the clear explanations provided by the agents.