Explainable Artificial Intelligence Agents for Data Cleaning and Pipeline Debugging in Realtime
摘要
Real-time data processing has become important in many fields such as healthcare, finance, and smart systems. Large volumes of data flow continuously through pipelines, where errors and inconsistencies often affect the final results. Data cleaning and pipeline debugging are needed to maintain data quality, but manual checking is slow and not suitable for real-time systems. This is where Explainable Artificial Intelligence (XAI) agents can help. XAI agents can point out the exact step in the pipeline where something went wrong. These agents not only clean the data but also give clear explanations of what went wrong and what changes were made. This helps data engineers and analysts fix issues quickly without going through every step manually. Many existing tools clean data in the background but do not explain what they do. This makes it hard for users to trust or improve the process. The proposed approach uses real-time XAI agents that detect data errors, fix them, and explain their actions using simple rules and examples. These agents also track the entire pipeline to find issues in logic or flow. Each action taken by the agent is logged with a reason so users can trace the changes. The system was tested on large real-time datasets from public and industrial sources. Evaluation metrics such as accuracy, error detection rate (EDR), and debugging time (DT) showed better performance when compared with traditional data cleaning tools. Users reported higher confidence in the results due to the clear explanations provided by the agents.