This paper presents a comprehensive comparative analysis of machine learning algorithm performance under different parallel computing architectures, specifically examining MPI-based CPU parallelization and GPU acceleration strategies. We evaluate five widely-used algorithms – linear regression, polynomial regression, decision tree, random forest, and support vector regression – across three datasets of varying scales: small (2,939 samples, 25 features), medium (20,640 samples, 8 features), and large (80,000 samples, 13 features). Systematic performance evaluation was conducted using 1, 2, 4, and 8 parallel processes on both CPU and GPU platforms, measuring accuracy metrics (R2, MSE, MAE) alongside training time. Our results demonstrate that Random Forest and Support Vector Regression exhibit superior scalability and accuracy on GPU architectures, particularly for large-scale datasets where they achieve R2 values approaching 1.0. Notably, GPU-based parallel computing provides substantial improvements in prediction accuracy due to enhanced processing of larger data volumes compared to CPU implementations. The findings indicate that hybrid MPI-GPU computing systems can deliver significant performance gains for machine learning workloads, with optimal configurations varying by algorithm type and dataset characteristics. This study provides practical guidelines for selecting appropriate parallel computing strategies based on data scale and algorithm requirements, contributing to more efficient deployment of machine learning systems in high-performance computing environments.