Scholarlink

<p>Due to unprecedented performance, large-scale language models (LLMs) have become widespread in various fields. Researchers have developed many heuristic parallelism strategies to facilitate training in high-performance computing platforms as LLMs grow too large to be trained on a single node. However, distributed deep learning performance models suffer from inaccurate and time-consuming problems to help determine optimal strategies. To solve this, we propose ParallelSim, an accurate, generic, and efficient simulator designed to estimate the performance of various strategies. ParallelSim can simulate distributed deep-learning programs written in PyTorch by converting the models into intermediate representation sub-graphs. To enhance ParallelSim’s accuracy, we analyze profiling errors and overlap from the perspective of GPU hardware design, adapting the profiling logic to more accurately model real-world scenarios. ParallelSim uses a hierarchical simulation engine that decouples the execution of inter-stage and intra-stage to accelerate the simulation. Finally, we evaluate ParallelSim on 16 DGX A100 nodes. The experimental results show that ParallelSim has an average prediction error of 1.83% on 16 nodes and helps to select the optimal strategies.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Parallelsim: an accurate, generic, and efficient simulator for distributed deep learning

  • 期刊论文

  • 发表日期：

    2026年3月16日

查看全文
  • Peng Liang,
  • Linbo Qiao,
  • Zhiquan Lai,
  • Dongsheng Li

摘要

Due to unprecedented performance, large-scale language models (LLMs) have become widespread in various fields. Researchers have developed many heuristic parallelism strategies to facilitate training in high-performance computing platforms as LLMs grow too large to be trained on a single node. However, distributed deep learning performance models suffer from inaccurate and time-consuming problems to help determine optimal strategies. To solve this, we propose ParallelSim, an accurate, generic, and efficient simulator designed to estimate the performance of various strategies. ParallelSim can simulate distributed deep-learning programs written in PyTorch by converting the models into intermediate representation sub-graphs. To enhance ParallelSim’s accuracy, we analyze profiling errors and overlap from the perspective of GPU hardware design, adapting the profiling logic to more accurately model real-world scenarios. ParallelSim uses a hierarchical simulation engine that decouples the execution of inter-stage and intra-stage to accelerate the simulation. Finally, we evaluate ParallelSim on 16 DGX A100 nodes. The experimental results show that ParallelSim has an average prediction error of 1.83% on 16 nodes and helps to select the optimal strategies.