Stream Processing is a very effective predominant paradigm for data processing. It provides an efficient approach to extract information from new data, as the data arrives. However, spikes in data throughput, can impact the accuracy and latency guarantees stream processing systems provide. This work proposes data sampling, a type of data reduction, as a solution to this problem. It provides a user-transparent implementation of two sampling methods in the Apache Spark Streaming framework. The results show a reduced amount of input data, leading to decreased processing time, but retaining a good accuracy in the extracted information.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Strainer: Windowing-Based Advanced Sampling in Stream Processing Systems

  • Nikola Koevski,
  • Sérgio Esteves,
  • Luís Veiga

摘要

Stream Processing is a very effective predominant paradigm for data processing. It provides an efficient approach to extract information from new data, as the data arrives. However, spikes in data throughput, can impact the accuracy and latency guarantees stream processing systems provide. This work proposes data sampling, a type of data reduction, as a solution to this problem. It provides a user-transparent implementation of two sampling methods in the Apache Spark Streaming framework. The results show a reduced amount of input data, leading to decreased processing time, but retaining a good accuracy in the extracted information.