Distributed Training

We propose OctoPipe, a novel pipeline parallelism system that reduces pipeline bubbles on heterogeneous models by co-optimizing partitioning, placement, and scheduling, achieving 1.22-2.14x throughput improvement over Megatron-LM.

We propose UniTG, the first unified system that fuses the LM and GNN phases of textual graph learning into a single end-to-end procedure, reducing learning makespan by up to 17.3x without compromising model quality.

A technical note on Helix, a compiler-based graph scheduling system that overlaps communication and computation for n-D model parallel training and inference.

A technical report on ResiHP, a resilient training system that detects fail-slow devices under noisy sequence-length variation and dynamically adapts 3D parallelism.

一篇关于 ResiHP 的技术报告：它在变长序列带来的噪声中识别 fail-slow 设备，并动态调整 3D 并行来提升大模型训练韧性。

Hybrid parallelism underpins large-scale LLM training across tens of thousands of GPUs. At such scale, hardware failures on individual …

Large Language Models (LLMs) have demonstrated impressive performance across various downstream tasks. When training these models, …