FbsPipe: Forward-Backward Separation Pipeline Parallelism Method for Deep Learning
摘要
To reduce communication overhead during deep neural network (DNN) training, existing pipeline methods typically store activation values locally or recompute them as needed. With the rapid advancement of hardware architectures, modern systems provide increasingly higher communication bandwidth. This creates an opportunity to exploit bandwidth resources for alleviating the growing challenges of computation and memory efficiency in large-scale training. To this end, this paper proposes FbsPipe, a forward–backward separation pipeline training method specifically designed for high-bandwidth environments. By decoupling forward and backward computations and leveraging high-bandwidth interconnects, FbsPipe effectively reduces both computation and memory overhead, thereby improving pipeline performance. Experimental results across models of varying scales demonstrate that FbsPipe consistently achieves higher throughput and lower memory consumption, with throughput improvements of up to 49%. These findings highlight FbsPipe as an efficient and scalable solution for training large-scale models on next-generation highly interconnected, high-bandwidth architectures such as system-on-wafer and wafer-scale systems.