{"categories":[{"label":"Software Development","url":"https://skillfed.io/packages/category/software-development/17"},{"label":"Libraries","url":"https://skillfed.io/packages/category/software-development-libraries/10"},{"label":"Scientific/Engineering","url":"https://skillfed.io/packages/category/scientific-engineering/8"},{"label":"Python Modules","url":"https://skillfed.io/packages/category/software-development-libraries-python-modules/21"},{"label":"Artificial Intelligence","url":"https://skillfed.io/packages/category/scientific-engineering-artificial-intelligence/10"}],"enrichment":{"capability":"InstantTensor provides a high-throughput Safetensors loader optimized for moving model weights from disk to GPU memory, with support for distributed loading via torch.distributed.","skillfed_tags":["gpu-io","model-loading","distributed-inference"],"use_cases":["Loading large language models (30B+) onto single or multi-GPU setups where cold-start latency matters.","Serving scenarios where host memory is constrained by KV cache offloading or other allocations.","Multi-model serving where models are switched frequently and cannot be cached together.","Distributed inference with tensor parallelism (TP=8+) where each GPU receives small, non-contiguous shards.","Loading model checkpoints from tmpfs or high-bandwidth storage (\u22655 GB/s) where direct I/O is beneficial."],"what_it_does":"InstantTensor is a Safetensors loader built to maximize I/O throughput when loading model weights onto GPU. It uses direct I/O, tuned concurrency, and pipelining to avoid slow page cache allocation, and supports distributed loading via torch.distributed NCCL for coordinated multi-GPU reads. The package is designed for scenarios where models are large, storage bandwidth is high, or the model cannot be cached in host memory\u2014such as when memory is consumed by KV cache offloading in LLM serving, or when loading multiple models that cannot fit simultaneously.\n\nThe library exposes a `safe_open` context manager that yields tensors from Safetensors files, with options for zero-copy streaming into preallocated buffers, backend selection (AIO, URING, CUFILE, MMAP), and buffered vs. direct I/O modes. It integrates with torch and requires a GPU platform (CUDA or ROCm). The package is in alpha status and depends only on torch.","worth_installing":"Yes, if you load large Safetensors models onto GPU and have either high storage bandwidth, constrained host memory, or frequent model switching. The package is actively maintained, permissively licensed, and already integrated into vLLM. Install friction is moderate (compiled wheels, torch dependency). Not recommended if you load small models infrequently or have ample host memory for caching\u2014standard Safetensors loading will suffice."},"id":"instanttensor","links":{"html":"https://skillfed.io/packages/instanttensor","md":"https://skillfed.io/packages/instanttensor.md","pypi":"https://pypi.org/project/instanttensor/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-05-27","license_spdx":null,"license_treatment":"permissive","name":"instanttensor","python_support":"supports_current","summary":"An ultra-fast, distributed Safetensors loader"},"popularity":{"monthly_downloads":92667,"position":13435,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.1.9"}