Scholarlink

With the rapid development of deep learning, the training of large-scale pre-trained models has placed increasingly high demands on computing infrastructure. However, due to limitations in cost and resources, high-performance computing clusters remain inaccessible to most enterprise researchers. Integrating heterogeneous computing resources distributed across the world regions is gradually becoming a viable alternative. Nevertheless, these approaches face the dual challenges of heterogeneity in both network and computing devices. To address these challenges, we introduce HarmonyTrain, a bespoke distributed training paradigm tailored for dual-heterogeneous environments. Initially, we design a novel hybrid parallel strategy PatchPipe to effectively balance the performance differences among heterogeneous devices. Following this, we develop a model partitioning and task placement algorithm to generate efficient parallel training plans. In addition, to tackle the straggler problem in heterogeneous environments, we propose a semi-synchronous gradient synchronization with adaptive weighting (SSAW) mechanism to improve training efficiency. The experimental results show that, with the same computing resources, our method achieves a performance improvement of 22% to 63.1% compared to baseline approaches.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

HarmonyTrain: A Hybrid Parallel Training Method for Deep Learning Models in Dual-Heterogeneous Environments

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Jintao Li,
  • Chuantao Li,
  • Chunxiao Wang,
  • Zhigang Zhao,
  • Guangdong Zhang,
  • Zhiwei Zhou,
  • Yuxin Wang,
  • Yan Xu,
  • Fulai Liu,
  • Anbang Li

摘要

With the rapid development of deep learning, the training of large-scale pre-trained models has placed increasingly high demands on computing infrastructure. However, due to limitations in cost and resources, high-performance computing clusters remain inaccessible to most enterprise researchers. Integrating heterogeneous computing resources distributed across the world regions is gradually becoming a viable alternative. Nevertheless, these approaches face the dual challenges of heterogeneity in both network and computing devices. To address these challenges, we introduce HarmonyTrain, a bespoke distributed training paradigm tailored for dual-heterogeneous environments. Initially, we design a novel hybrid parallel strategy PatchPipe to effectively balance the performance differences among heterogeneous devices. Following this, we develop a model partitioning and task placement algorithm to generate efficient parallel training plans. In addition, to tackle the straggler problem in heterogeneous environments, we propose a semi-synchronous gradient synchronization with adaptive weighting (SSAW) mechanism to improve training efficiency. The experimental results show that, with the same computing resources, our method achieves a performance improvement of 22% to 63.1% compared to baseline approaches.