From Big Data to Smart Data: Effective Subset Generation for Machine Learning Model Training
摘要
In machine learning, the rapid growth of data volume and model complexity significantly enhances performance but escalates computational costs and resource demands. In order to improve training efficiency and reduce computational cost, a high-value data subset can be subtracted from the original dataset after data value evaluation. Therefore, an effective subset generation method based on data value evaluation is proposed. In this method, a probability model is built first. Then, the value of the data is obtained. Finally, the required number of high-value subsets is generated. In this way, the data size is reduced while the model’s performance loss is limited. Two experiments are designed to verify the effectiveness. The first experiment is to prove the effectiveness of our method and indicate that it can reduce data size while preserving model performance. We segment the dataset into 24 subsets, sort by value evaluation and assess their predictive accuracy, comparing outcomes to the entire dataset. Results indicate that predictions from 24 data subsets can closely match those made using the entire dataset. The second experiment is to prove that subsets generated by our method outperform random sorted subsets. We segment the dataset using our method and generate 33 subsets. We analyze their prediction performance and test against randomly sorted subsets. Results demonstrate that subsets sorted by data value show superior model training outcomes compared to randomly split subsets. The proposed method for generating data subsets based on data value shows advantages in enhancing the efficiency of machine learning model training.