instanttensor

What it is and what it does

InstantTensor is a Safetensors loader built to maximize I/O throughput when loading model weights onto GPU. It uses direct I/O, tuned concurrency, and pipelining to avoid slow page cache allocation, and supports distributed loading via torch.distributed NCCL for coordinated multi-GPU reads. The package is designed for scenarios where models are large, storage bandwidth is high, or the model cannot be cached in host memory—such as when memory is consumed by KV cache offloading in LLM serving, or when loading multiple models that cannot fit simultaneously.

The library exposes a `safe_open` context manager that yields tensors from Safetensors files, with options for zero-copy streaming into preallocated buffers, backend selection (AIO, URING, CUFILE, MMAP), and buffered vs. direct I/O modes. It integrates with torch and requires a GPU platform (CUDA or ROCm). The package is in alpha status and depends only on torch.

Use it for:

• Loading large language models (30B+) onto single or multi-GPU setups where cold-start latency matters.
• Serving scenarios where host memory is constrained by KV cache offloading or other allocations.
• Multi-model serving where models are switched frequently and cannot be cached together.
• Distributed inference with tensor parallelism (TP=8+) where each GPU receives small, non-contiguous shards.
• Loading model checkpoints from tmpfs or high-bandwidth storage (≥5 GB/s) where direct I/O is beneficial.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

InstantTensor provides a high-throughput Safetensors loader optimized for moving model weights from disk to GPU memory, with support for distributed loading via torch.distributed.

Yes, if you load large Safetensors models onto GPU and have either high storage bandwidth, constrained host memory, or frequent model switching. The package is actively maintained, permissively licensed, and already integrated into vLLM. Install friction is moderate (compiled wheels, torch dependency). Not recommended if you load small models infrequently or have ample host memory for caching—standard Safetensors loading will suffice.

Install

instanttensor on PyPI

pip install instanttensor

uv add instanttensor

poetry add instanttensor

Installing instanttensor

Before you install

Medium friction: compiled wheels available for Python 3.10–3.14 on Linux x86_64, but requires torch as a runtime dependency. Active maintenance status with recent release.

License in practice

Apache License 2.0 (permissive): you may use, modify, and distribute freely provided you include a copy of the license and document any changes. No restrictions on commercial use.

Quickstart

pip install instanttensor

from instanttensor import safe_open

with safe_open("model.safetensors", framework="pt", device=0) as f:
    for name, tensor in f.tensors():
        print(name, tensor.shape)

Requires CUDA or ROCm GPU and torch installed; Linux x86_64 only in current wheels.

Verify before relying

• Whether zero-copy mode (copy=False) is production-ready or still experimental given alpha status.
• Performance gains on non-H200/H100 hardware or with smaller models.
• Stability of distributed loading with subgroups across different parallelism strategies.

Package facts

Evidence: instanttensor-0.1.9-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl; instanttensor-0.1.9-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl; instanttensor-0.1.9-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl; instanttensor-0.1.9-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl; instanttensor-0.1.9-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Tags

More Software Development packages