torchft-nightly

What it is and what it does

torchft-nightly is a PyTorch library that adds fault tolerance to distributed training by detecting and recovering from worker failures at the training step level, rather than requiring full job restart. It provides coordination primitives, fault-tolerant process groups, and checkpoint transports built on a Lighthouse server that tracks worker health via heartbeating. The library includes out-of-the-box algorithms for Fault Tolerant DDP, Fault Tolerant HSDP (combining FSDP, tensor parallelism, and DDP), LocalSGD, and DiLoCo.

The package depends on torch, opentelemetry-api, opentelemetry-sdk, and opentelemetry-exporter-otlp-proto-http for observability. It is designed for large-scale training scenarios where worker failures are common and restarting the entire job is expensive. Integration requires wrapping your model, optimizer, and process group with torchft's Manager and fault-tolerant wrappers, then running a Lighthouse server to coordinate membership changes across replica groups.

Use it for:

• Training large models like Llama 3 70B on multi-node clusters where worker failures are expected and full restart is costly.
• Implementing fault-tolerant DDP training loops that gracefully handle node failures without losing training progress.
• Scaling HSDP training across heterogeneous hardware with automatic recovery from individual worker outages.
• Running parameter server training with reconfigurable process groups that adapt to membership changes.
• Distributed training on preemptible or spot instances where interruptions are frequent and expected.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides per-step fault tolerance for PyTorch distributed training, allowing jobs to recover from worker failures without restarting the entire training process.

Yes, with conditions. Install if you are training large distributed models on infrastructure where worker failures are common and you need per-step recovery without full job restart. The active maintenance, recent releases, and zero known vulnerabilities are positive signals. However, verify the BSD 3-Clause license applies to your use case (metadata is unclear), and be aware that medium install friction (Rust compilation, system dependencies) and nightly-only availability mean this is best suited for teams with infrastructure expertise rather than casual users.

Install

torchft-nightly on PyPI

pip install torchft-nightly

uv add torchft-nightly

poetry add torchft-nightly

Installing torchft-nightly

Before you install

Medium install friction due to compiled Rust components and system dependencies (protobuf-compiler, Rust toolchain). Active maintenance with recent releases; nightly builds available. Requires PyTorch 2.7 RC+ or Nightly.

License in practice

License treatment is unclear; the repository indicates BSD 3-Clause licensing, but this is not formally declared in package metadata. Verify licensing terms before use in proprietary projects.

Quickstart

pip install torchft-nightly

from torchft import Manager, DistributedDataParallel, Optimizer, ProcessGroupGloo

manager = Manager(pg=ProcessGroupGloo(), load_state_dict=..., state_dict=...)
m = DistributedDataParallel(manager, model)
optimizer = Optimizer(manager, optim.AdamW(m.parameters()))

Requires Rust, protobuf-compiler, and PyTorch 2.7 RC+ or Nightly. Lighthouse server must be running for DDP/HSDP coordination (e.g., torchft_lighthouse --min_replicas 1).

Verify before relying

• Whether BSD 3-Clause license is formally enforced or if there are additional licensing constraints not reflected in metadata.
• Stability guarantees for nightly builds and whether production use is recommended.
• Performance overhead of per-step fault tolerance compared to standard training without recovery.

Package facts

Evidence: torchft_nightly-2026.8.14-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; torchft_nightly-2026.8.14-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; torchft_nightly-2026.8.14-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; torchft_nightly-2026.8.14-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; torchft_nightly-2026.8.14-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Tags

More Distributed Computing packages