---
id: torchft-nightly
version: "2026.8.14"
license: unclear
license_treatment: unclear
maintenance: active
---

# torchft-nightly

License: unclear · Maintenance: active · Downloads: 102.9K/mo

## What it is and what it does

torchft-nightly is a PyTorch library that adds fault tolerance to distributed training by detecting and recovering from worker failures at the training step level, rather than requiring full job restart. It provides coordination primitives, fault-tolerant process groups, and checkpoint transports built on a Lighthouse server that tracks worker health via heartbeating. The library includes out-of-the-box algorithms for Fault Tolerant DDP, Fault Tolerant HSDP (combining FSDP, tensor parallelism, and DDP), LocalSGD, and DiLoCo.

The package depends on torch, opentelemetry-api, opentelemetry-sdk, and opentelemetry-exporter-otlp-proto-http for observability. It is designed for large-scale training scenarios where worker failures are common and restarting the entire job is expensive. Integration requires wrapping your model, optimizer, and process group with torchft's Manager and fault-tolerant wrappers, then running a Lighthouse server to coordinate membership changes across replica groups.

Use it for:

- Training large models like Llama 3 70B on multi-node clusters where worker failures are expected and full restart is costly.
- Implementing fault-tolerant DDP training loops that gracefully handle node failures without losing training progress.
- Scaling HSDP training across heterogeneous hardware with automatic recovery from individual worker outages.
- Running parameter server training with reconfigurable process groups that adapt to membership changes.
- Distributed training on preemptible or spot instances where interruptions are frequent and expected.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Provides per-step fault tolerance for PyTorch distributed training, allowing jobs to recover from worker failures without restarting the entire training process.

Yes, with conditions. Install if you are training large distributed models on infrastructure where worker failures are common and you need per-step recovery without full job restart. The active maintenance, recent releases, and zero known vulnerabilities are positive signals. However, verify the BSD 3-Clause license applies to your use case (metadata is unclear), and be aware that medium install friction (Rust compilation, system dependencies) and nightly-only availability mean this is best suited for teams with infrastructure expertise rather than casual users.

## Install

pip install torchft-nightly
uv add torchft-nightly
poetry add torchft-nightly

## Installing torchft-nightly

Before you install: Medium install friction due to compiled Rust components and system dependencies (protobuf-compiler, Rust toolchain). Active maintenance with recent releases; nightly builds available. Requires PyTorch 2.7 RC+ or Nightly.

License in practice: License treatment is unclear; the repository indicates BSD 3-Clause licensing, but this is not formally declared in package metadata. Verify licensing terms before use in proprietary projects.

Quickstart:

pip install torchft-nightly

from torchft import Manager, DistributedDataParallel, Optimizer, ProcessGroupGloo

manager = Manager(pg=ProcessGroupGloo(), load_state_dict=..., state_dict=...)
m = DistributedDataParallel(manager, model)
optimizer = Optimizer(manager, optim.AdamW(m.parameters()))

Requires Rust, protobuf-compiler, and PyTorch 2.7 RC+ or Nightly. Lighthouse server must be running for DDP/HSDP coordination (e.g., torchft_lighthouse --min_replicas 1).

Verify before relying:

- Whether BSD 3-Clause license is formally enforced or if there are additional licensing constraints not reflected in metadata.
- Stability guarantees for nightly builds and whether production use is recommended.
- Performance overhead of per-step fault tolerance compared to standard training without recovery.

## Package facts

- License: not declared (unclear)
- Python support: supports_current
- Install friction: medium
- Maintenance: active
- Downloads: 102.9K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

pytorch fault tolerance, distributed training recovery, ddp fault tolerance, hsdp fault tolerance, pytorch training resilience, worker failure recovery, distributed training checkpointing, distributed-training, fault-tolerance, pytorch

[View on SkillFed](https://skillfed.io/packages/torchft-nightly) · [View on PyPI](https://pypi.org/project/torchft-nightly/)