LLM Training

A technical note on Helix, a compiler-based graph scheduling system that overlaps communication and computation for n-D model parallel training and inference.

基于我们的 ACM Computing Surveys 论文，梳理 GPU 数据中心里的训练、推理、HPO、混合负载以及未来调度器设计。

A technical report on ResiHP, a resilient training system that detects fail-slow devices under noisy sequence-length variation and dynamically adapts 3D parallelism.

一篇关于 ResiHP 的技术报告：它在变长序列带来的噪声中识别 fail-slow 设备，并动态调整 3D 并行来提升大模型训练韧性。

A technical story behind Hydro’s Bubble Squeezer, which runs surrogate hyperparameter tuning trials inside the idle bubbles of pipeline-parallel large-model training.

A technical guide to GPU datacenter scheduling based on our ACM Computing Surveys paper, covering training, inference, HPO, mixed workloads, and future scheduler design.

Hybrid parallelism underpins large-scale LLM training across tens of thousands of GPUs. At such scale, hardware failures on individual …

Large Language Models (LLMs) have presented impressive performance across several transformative tasks. However, it is non-trivial to …

Large Language Models (LLMs) have demonstrated impressive performance across various downstream tasks. When training these models, …

Hyperparameter tuning is an essential step in deep learning model development that provides better model performance at the cost of …