Fault Tolerance

A technical report on ResiHP, a resilient training system that detects fail-slow devices under noisy sequence-length variation and dynamically adapts 3D parallelism.

一篇关于 ResiHP 的技术报告：它在变长序列带来的噪声中识别 fail-slow 设备，并动态调整 3D 并行来提升大模型训练韧性。

A technical note on GPU checkpoint/restore for schedulers, using FlowGPU as the main reference and my cudaw prototype as the first version of the codebase.

GPU checkpointing and restore promises to enable emerging tasks, such as deep learning, to benefit from functionalities like task …

Hybrid parallelism underpins large-scale LLM training across tens of thousands of GPUs. At such scale, hardware failures on individual …