Big data has become essential in fields such as geospatial analysis and disaster prediction, where it enhances the accuracy and efficiency of predictive models. The Support Vector Machine (SVM) algorithm is widely used for such tasks, but its application to large-scale datasets faces challenges, including excessive deviation in subsets distribution, insufficient parallel training performance, and poor filtering of non-support vectors. To overcome the above limitations, a parallel SVM algorithm based on MapReduce (PSVM-MR) is proposed in this paper, which contains two parts: data partition and parallel SVM training. First, a data partition method based on relative entropy (DP-RE) is proposed, which calculates the relative entropy to avoid excessive deviation of subsets distribution. Next, a redundancy level removing method based on cosine similarity (RLR-CS) is presented to address the insufficient performance of parallel training by removing the redundancy levels in the cascade structure. Finally, a non-support vector filtering method (NSVF) is proposed, which improves the capability of non-support vector filtering by combining rough identification and singular vector identification. Experiment shows that the proposed algorithm has lower training costs and higher parallel efficiency than the general parallel SVM algorithm.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

PSVM-MR: A Parallel Support Vector Machine Algorithm Based on MapReduce

  • Bin-bin Guo,
  • Yimin Mao,
  • A Yaser,
  • Neelakandan Chandrasekaran,
  • Le Kang,
  • Wenhao Li,
  • Decheng Miao

摘要

Big data has become essential in fields such as geospatial analysis and disaster prediction, where it enhances the accuracy and efficiency of predictive models. The Support Vector Machine (SVM) algorithm is widely used for such tasks, but its application to large-scale datasets faces challenges, including excessive deviation in subsets distribution, insufficient parallel training performance, and poor filtering of non-support vectors. To overcome the above limitations, a parallel SVM algorithm based on MapReduce (PSVM-MR) is proposed in this paper, which contains two parts: data partition and parallel SVM training. First, a data partition method based on relative entropy (DP-RE) is proposed, which calculates the relative entropy to avoid excessive deviation of subsets distribution. Next, a redundancy level removing method based on cosine similarity (RLR-CS) is presented to address the insufficient performance of parallel training by removing the redundancy levels in the cascade structure. Finally, a non-support vector filtering method (NSVF) is proposed, which improves the capability of non-support vector filtering by combining rough identification and singular vector identification. Experiment shows that the proposed algorithm has lower training costs and higher parallel efficiency than the general parallel SVM algorithm.