Stream processing engines (SPEs) have emerged as critical components for real-time data analytics, leveraging the scalability of cloud computing. Despite their widespread adoption, existing SPEs face performance bottlenecks due to inefficient utilization of single-node memory bandwidth and high communication overhead in distributed systems. This paper introduces a hybrid optimization framework that integrates vertical and horizontal scaling strategies to address these challenges. Inter-process communication overhead is eliminated by replacing traditional channels with shared-memory buffers, enabling seamless data exchange within nodes. Additionally, the number of actors is minimized through operator fusion, and a non-blocking global store is introduced to enhance data synchronization without sacrificing consistency. Leveraging NUMA-aware processing and SIMD parallelism, resource utilization is further optimized within single nodes while maintaining scalability across multiple nodes. Experimental results demonstrate significant performance improvements in throughput and latency, validating the proposed framework’s ability to balance compute efficiency with distributed scalability. This approach paves the way for future enhancements in cloud-based stream processing platforms, enabling more responsive and resource-efficient analytics pipelines.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Hybrid Optimization Strategies for High-Throughput Stream Processing Engines in Cloud Environments

  • Shubham Malhotra

摘要

Stream processing engines (SPEs) have emerged as critical components for real-time data analytics, leveraging the scalability of cloud computing. Despite their widespread adoption, existing SPEs face performance bottlenecks due to inefficient utilization of single-node memory bandwidth and high communication overhead in distributed systems. This paper introduces a hybrid optimization framework that integrates vertical and horizontal scaling strategies to address these challenges. Inter-process communication overhead is eliminated by replacing traditional channels with shared-memory buffers, enabling seamless data exchange within nodes. Additionally, the number of actors is minimized through operator fusion, and a non-blocking global store is introduced to enhance data synchronization without sacrificing consistency. Leveraging NUMA-aware processing and SIMD parallelism, resource utilization is further optimized within single nodes while maintaining scalability across multiple nodes. Experimental results demonstrate significant performance improvements in throughput and latency, validating the proposed framework’s ability to balance compute efficiency with distributed scalability. This approach paves the way for future enhancements in cloud-based stream processing platforms, enabling more responsive and resource-efficient analytics pipelines.