A Hybrid Architecture for Real-Time Data Ingestion and Querying Using Kafka and Intelligent Database Systems
摘要
Rapid expansion of data in digital ecosystems today demands scalable and efficient architectures for real-time ingestion and analytics. In this paper, we propose and design a hybrid architecture deployed by Apache Kafka and intelligent data systems, which support on-the-fly big data ingestions, computations and query processing. The system is based on using distributed stream processing with Kafka for high-throughput ingestion of data into intelligent databases like PostgreSQL with AI-based indexing and query optimization for context-aware querying at low-latencies. In this work, we present a modular pipeline that enables schema evolution, fault-tolerance, and intelligent workload management. Experiments show that we achieved high performance gains on both data ingestion speed, query response latency and system scalability over the traditional ETL based solutions. This combination architecture forms the basis for mission-critical applications that require real-time insights, such as IoT analytics, financial monitoring and healthcare informatics applications.