Information management challenges the use of cloud-native enterprise platforms like Salesforce and Workday, which have become very widespread in the financial services business. But, because these SaaS ecosystems lack real-time, compliant, and explainable analytics, it becomes a critical issue for regulated industries governed by the GDPR, SOX/HOX, and HIPAA frameworks. The proposed work presents a fully scalable and open-source data engineering solution based on Apache Kafka, Apache Spark, Hudi, and Airflow to enable integrated compliance enforcement of AI-driven real-time analytics. To test the effectiveness of the constructed framework, a high-frequency synthetic dataset was developed to create event flow, as recommended by Salesforce, such as leading conversion, transactional logs, and misuse with anomalous logins. The machine learning models to be implemented included XG-Boost, AutoML, and Isolation Forest embedded in the pipeline to detect fraud, validate the KYC, and identify anomalies. Assessment criteria show that XGBoost delivered 92% accuracy within sub-250 ms of latency and more than 25,000 active events per second. Schema validation, audit logging, and encryption at rest were all enforced in the architecture effectively, demonstrating conformance with enterprise governance policies. The study paves the way towards coupling streaming analytics and regulatory compliance by making live, explainable AI part of the cloud-based workflow. Its modular, vendor-neutral design allows time-travel audits, rollback recovery, and role-based access controls. The results can be allocated to the growing RegTech body of knowledge to provide much-needed guidance and a convenient, scalable blueprint for implementing an AI-driven system in a financial or highly compliant sector.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Bridging Data and AI: A Scalable Open-Source Framework for Real-Time Analytics in Regulated Financial Platforms

  • Gokulram Krishnan,
  • Aditya Ramaswamy,
  • Binita Mukesh Shah

摘要

Information management challenges the use of cloud-native enterprise platforms like Salesforce and Workday, which have become very widespread in the financial services business. But, because these SaaS ecosystems lack real-time, compliant, and explainable analytics, it becomes a critical issue for regulated industries governed by the GDPR, SOX/HOX, and HIPAA frameworks. The proposed work presents a fully scalable and open-source data engineering solution based on Apache Kafka, Apache Spark, Hudi, and Airflow to enable integrated compliance enforcement of AI-driven real-time analytics. To test the effectiveness of the constructed framework, a high-frequency synthetic dataset was developed to create event flow, as recommended by Salesforce, such as leading conversion, transactional logs, and misuse with anomalous logins. The machine learning models to be implemented included XG-Boost, AutoML, and Isolation Forest embedded in the pipeline to detect fraud, validate the KYC, and identify anomalies. Assessment criteria show that XGBoost delivered 92% accuracy within sub-250 ms of latency and more than 25,000 active events per second. Schema validation, audit logging, and encryption at rest were all enforced in the architecture effectively, demonstrating conformance with enterprise governance policies. The study paves the way towards coupling streaming analytics and regulatory compliance by making live, explainable AI part of the cloud-based workflow. Its modular, vendor-neutral design allows time-travel audits, rollback recovery, and role-based access controls. The results can be allocated to the growing RegTech body of knowledge to provide much-needed guidance and a convenient, scalable blueprint for implementing an AI-driven system in a financial or highly compliant sector.