LLM Training

We propose OctoPipe, a novel pipeline parallelism system that reduces pipeline bubbles on heterogeneous models by co-optimizing partitioning, placement, and scheduling, achieving 1.22-2.14x throughput improvement over Megatron-LM.

A technical note on Helix, a compiler-based graph scheduling system that overlaps communication and computation for n-D model parallel training and inference.

A technical report on ResiHP, a resilient training system that detects fail-slow devices under noisy sequence-length variation and dynamically adapts 3D parallelism.

一篇关于 ResiHP 的技术报告：它在变长序列带来的噪声中识别 fail-slow 设备，并动态调整 3D 并行来提升大模型训练韧性。

A technical story behind Hydro’s Bubble Squeezer, which runs surrogate hyperparameter tuning trials inside the idle bubbles of pipeline-parallel large-model training.

一篇关于 Hydro Bubble Squeezer 的技术文章：它把轻量级超参数搜索任务放进大模型流水线并行训练的空泡里运行。

A technical guide to GPU datacenter scheduling based on our ACM Computing Surveys paper, covering training, inference, HPO, mixed workloads, and future scheduler design.

Hybrid parallelism underpins large-scale LLM training across tens of thousands of GPUs. At such scale, hardware failures on individual …

Large Language Models (LLMs) have presented impressive performance across several transformative tasks. However, it is non-trivial to …

Large Language Models (LLMs) have demonstrated impressive performance across various downstream tasks. When training these models, …