Handling Missing Data in Real-Time Big Data Using the Linear-Blend Approach
摘要
Missing data in real-time systems can introduce significant risks, particularly in the context of big data where high velocity and volume amplify these challenges. Real-time applications such as live video streaming or sensor-based machinery monitoring often cannot afford the omission of missing data, and traditional machine learning approaches like deep learning or probabilistic models may not be feasible due to their computational demands. The aim is to develop a simple, linear, and effective method for real-time data imputation. This paper proposes the Linear-Blend approach, integrating Mean Imputation, Last Observation Carried Forward (LOCF), and novel Delta-Mode Imputation to balance accuracy and efficiency in real-time data streams. The Linear-Blend method achieves linear time complexity and constant space complexity, making it suitable for high-velocity environments while maintaining high imputation accuracy without the complexity of neural network-based models. Experimental results demonstrate that the proposed approach provides a practical solution for handling missing data in real-time big data systems, offering superior performance compared to more computationally intensive methods.