<p>In today’s expansive big data computing environments, intricate jobs are partitioned into numerous sub-tasks executed concurrently, resulting in accelerated job completion and reduced energy consumption. However, the emergence of straggler tasks, which exhibit atypical execution times compared to their counterparts, poses a significant challenge. Stragglers are infrequent but can substantially impede data center performance, causing delays, reduced efficiency, and overall processing bottlenecks. Proactively identifying and accurately predicting stragglers is imperative for data center administrators, enabling timely intervention to mitigate their impact. Existing machine learning methods encounter difficulties in straggler prediction due to their dependence on complete labels and assumptions about latency distributions. These assumptions often fall short of reflecting data center environments’ dynamic and unpredictable nature. To address these challenges, this article proposes an innovative solution employing the Long Short-Term Memory (LSTM) model for utilizing straggler estimation in Spark jobs. The LSTM network efficiently identifies and optimizes straggler tasks using sequential data processing to predict the time left to complete activities. To assess the robustness of the LSTM network, this article utilizes a merged dataset compiled from both Google and Alibaba traces. This unique amalgamation of data traces from two industry giants enriches our research and strengthens our approach’s real-world applicability. The LSTM-based solution excels in detecting straggler tasks and exhibits promising potential to optimize parallel data processing, significantly enhancing efficiency and performance in modern data center environments.</p>