Efficient data loading is a critical component of high-performance machine learning workflows, particularly when working with large-scale datasets. This study benchmarks the performance of sequential versus parallel data loading in PyTorch using a synthetic classification dataset and a simple Multi-Layer Perceptron (MLP) model. Dataset sizes ranged from 100,000 to 5,000,000 samples, tested in increments of 100,000. Two data loading configurations were compared: sequential loading (num_workers = 0) and parallel loading (num_workers = 2). The experiment measured key performance metrics, including total data loading time, total training epoch time, and the resulting speedup from parallel loading. Results show that parallel data loading significantly reduces data I/O overhead, achieving speedups of up to 1.78× for large datasets. The findings highlight the scalability and efficiency gains of parallel data loaders and underscore their importance in optimizing training pipelines for data-intensive machine learning applications.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Evaluating the Impact of Parallel Data Loading on Training Performance in PyTorch

  • Mirela Sino,
  • Ervin Domazet

摘要

Efficient data loading is a critical component of high-performance machine learning workflows, particularly when working with large-scale datasets. This study benchmarks the performance of sequential versus parallel data loading in PyTorch using a synthetic classification dataset and a simple Multi-Layer Perceptron (MLP) model. Dataset sizes ranged from 100,000 to 5,000,000 samples, tested in increments of 100,000. Two data loading configurations were compared: sequential loading (num_workers = 0) and parallel loading (num_workers = 2). The experiment measured key performance metrics, including total data loading time, total training epoch time, and the resulting speedup from parallel loading. Results show that parallel data loading significantly reduces data I/O overhead, achieving speedups of up to 1.78× for large datasets. The findings highlight the scalability and efficiency gains of parallel data loaders and underscore their importance in optimizing training pipelines for data-intensive machine learning applications.