Ensuring compliance in data pipelines is crucial for data integrity, data security, and industry verticals’ regulatory compliance in areas such as health and finance. Traditional mechanisms for compliance checks have rested on manual auditing, which is time-consuming and prone to errors. In this paper, an AI-powered system for analyzing logs is designed, using machine learning models developed in Python to verify compliance. An HDFS log dataset found in the open domain is utilized for system testing and training, using Natural Language Processing and anomaly detection for categorizing logs. Experimental evaluation demonstrates the system to have 92.5% accuracy, 89.7% precision, and 91.2% sensitivity, better than standard rule-based systems. The result demonstrates AI's ability to improve monitoring for compliance, save time, and improve scalability. The real-time detection and implementation of blockchain for enhanced auditability will be undertaken in the future.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Simulating Compliance Checks in Data Pipelines Using AI-Powered Log Analysis with Python

  • Teja Krishna Kota

摘要

Ensuring compliance in data pipelines is crucial for data integrity, data security, and industry verticals’ regulatory compliance in areas such as health and finance. Traditional mechanisms for compliance checks have rested on manual auditing, which is time-consuming and prone to errors. In this paper, an AI-powered system for analyzing logs is designed, using machine learning models developed in Python to verify compliance. An HDFS log dataset found in the open domain is utilized for system testing and training, using Natural Language Processing and anomaly detection for categorizing logs. Experimental evaluation demonstrates the system to have 92.5% accuracy, 89.7% precision, and 91.2% sensitivity, better than standard rule-based systems. The result demonstrates AI's ability to improve monitoring for compliance, save time, and improve scalability. The real-time detection and implementation of blockchain for enhanced auditability will be undertaken in the future.