ManuMatic: Strategy Injection for Robust Automatic Hybrid Parallelism in Distributed DNN Training
摘要
Training modern deep neural networks (DNNs) requires hybrid parallelism. Automatic planners search data, tensor/model, and pipeline shardings with cost models, but decisions can drift from runtime optima due to framework/planner decoupling and overlap mis-modeling. We present ManuMatic, a light-touch planner that lets users pin a few critical operator shardings while automatically deriving globally consistent strategies for the rest. Inside a binary recursive partitioner, ManuMatic prioritizes pins via an infinite compromise price and decomposes multi-dimensional hints into two-way refinements; when hard constraints are infeasible, a soft-penalty variant applies. The design is profiling-free, preserves D-Rec’s short compilation time, and degenerates to D-Rec when no pins are given. Built atop D-Rec, ManuMatic delivers consistent speedups without cost-model reengineering: on Mixtral-8 \(\times \) 7B, an expert-parallel-aware BMM pin achieves 2.24 \(\times \) over D-Rec; on Llama3-8B, a sequence-parallel-aware MatMul pin reaches 2.04 \(\times \) ; on Qwen2.5-72B, a sequence-parallel-aware MatMul pin combined with BMPipe yields 1.45 \(\times \) over D-Rec and 1.30 \(\times \) over an expert plan. These results show that minimal guidance can robustify automatic parallelism while largely preserving automation.