Data reduction for big data application in IOT
摘要
The fast development of the Internet of Things (IoT) has cultivated the creation of large, heterogeneous, and high-dimensional data that have significant challenges in terms of effective storage, transmission, and processing. The traditional data-processing tools do not always work effectively in handling such large data, hence the rise in computational expenses and reduced responsiveness of the system. The paper overcomes the issue of data management in high-dimensional IoT-based big-data setting through the introduction of a hybrid dimensionality-reduction model that combines Principal Component Analysis (PCA) with t-Distributed Stochastic Neighbor Embedding (t-SNE). The structure utilizes linear feature-extraction ability of PCA and nonlinear manifold-learning of t-SNE to obtain better compression of data and inferring necessary structural information of data, thus enabling more scalable and reliable analytics of the IoT systems. The proposed framework was coded in Python and tested on the Kaggle Cats vs. Dogs image dataset, which is a typical example of high-dimensional IoT data. A preprocessing was performed followed by multi-rank dimensionality reduction and classification based on Support Vector Machines to evaluate the performance of the empirical investigation. It has been experimentally shown that a hybrid PCA-t-SNES generates a relative 5 -percent gain in data-reduction efficiency compared to conventional PCA, shrinking the original feature space by down-sampling 145,800 features to 450 dimensions with no significant variation in classification accuracy among varying levels of ranking (Rank-1 to Rank-32). The above results affirm that the introduced hybrid model is useful in reducing redundancy, maximizing computational efficiency, and maintaining the accuracy of the analysis. The study provides a reconfigurable and scalable base of IoT data analytics, and would dominate smart sensing, edge computing, and real-time decision-making systems.