Distributed Training

A technical note on Helix, a compiler-based graph scheduling system that overlaps communication and computation for n-D model parallel training and inference.

A technical report on ResiHP, a resilient training system that detects fail-slow devices under noisy sequence-length variation and dynamically adapts 3D parallelism.

一篇关于 ResiHP 的技术报告：它在变长序列带来的噪声中识别 fail-slow 设备，并动态调整 3D 并行来提升大模型训练韧性。

Hybrid parallelism underpins large-scale LLM training across tens of thousands of GPUs. At such scale, hardware failures on individual …

Large Language Models (LLMs) have demonstrated impressive performance across various downstream tasks. When training these models, …