---
id: instanttensor
version: "0.1.9"
license: Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for… (full text in the JSON record)
license_treatment: permissive
maintenance: active
---

# instanttensor — An ultra-fast, distributed Safetensors loader

License: permissive · Maintenance: active · Downloads: 92.7K/mo

## What it is and what it does

InstantTensor is a Safetensors loader built to maximize I/O throughput when loading model weights onto GPU. It uses direct I/O, tuned concurrency, and pipelining to avoid slow page cache allocation, and supports distributed loading via torch.distributed NCCL for coordinated multi-GPU reads. The package is designed for scenarios where models are large, storage bandwidth is high, or the model cannot be cached in host memory—such as when memory is consumed by KV cache offloading in LLM serving, or when loading multiple models that cannot fit simultaneously.

The library exposes a `safe_open` context manager that yields tensors from Safetensors files, with options for zero-copy streaming into preallocated buffers, backend selection (AIO, URING, CUFILE, MMAP), and buffered vs. direct I/O modes. It integrates with torch and requires a GPU platform (CUDA or ROCm). The package is in alpha status and depends only on torch.

Use it for:

- Loading large language models (30B+) onto single or multi-GPU setups where cold-start latency matters.
- Serving scenarios where host memory is constrained by KV cache offloading or other allocations.
- Multi-model serving where models are switched frequently and cannot be cached together.
- Distributed inference with tensor parallelism (TP=8+) where each GPU receives small, non-contiguous shards.
- Loading model checkpoints from tmpfs or high-bandwidth storage (≥5 GB/s) where direct I/O is beneficial.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

InstantTensor provides a high-throughput Safetensors loader optimized for moving model weights from disk to GPU memory, with support for distributed loading via torch.distributed.

Yes, if you load large Safetensors models onto GPU and have either high storage bandwidth, constrained host memory, or frequent model switching. The package is actively maintained, permissively licensed, and already integrated into vLLM. Install friction is moderate (compiled wheels, torch dependency). Not recommended if you load small models infrequently or have ample host memory for caching—standard Safetensors loading will suffice.

## Install

pip install instanttensor
uv add instanttensor
poetry add instanttensor

## Installing instanttensor

Before you install: Medium friction: compiled wheels available for Python 3.10–3.14 on Linux x86_64, but requires torch as a runtime dependency. Active maintenance status with recent release.

License in practice: Apache License 2.0 (permissive): you may use, modify, and distribute freely provided you include a copy of the license and document any changes. No restrictions on commercial use.

Quickstart:

pip install instanttensor

from instanttensor import safe_open

with safe_open("model.safetensors", framework="pt", device=0) as f:
    for name, tensor in f.tensors():
        print(name, tensor.shape)

Requires CUDA or ROCm GPU and torch installed; Linux x86_64 only in current wheels.

Verify before relying:

- Whether zero-copy mode (copy=False) is production-ready or still experimental given alpha status.
- Performance gains on non-H200/H100 hardware or with smaller models.
- Stability of distributed loading with subgroups across different parallelism strategies.

## Package facts

- License: Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for… (full text in the JSON record) (permissive)
- Python support: supports_current
- Install friction: medium
- Maintenance: active
- Downloads: 92.7K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

safetensors gpu loader, fast model weight loading, distributed tensor loading, pytorch model io optimization, gpu direct storage, model checkpoint streaming, gpu-io, model-loading, distributed-inference

[View on SkillFed](https://skillfed.io/packages/instanttensor) · [View on PyPI](https://pypi.org/project/instanttensor/)