mooncake-transfer-engine
A KVCache-centric Disaggregated Architecture for large-scale LLM inference and training.
Decision gist · record as of 2026-08-14
Yes, if you are building or deploying a disaggregated LLM inference or training system at scale and your framework (vLLM, SGLang, TensorRT LLM, LMDeploy) has native Mooncake support. Do not install directly for single-machine use; it is an infrastructure component for multi-node GPU clusters. Active maintenance, no known vulnerabilities, and permissive license make it safe to adopt in production.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Linux (manylinux_2_28), Python 3.10 or later, and NVIDIA CUDA environment; designed for GPU cluster deployment, not single-machine use.
- Medium install friction due to compiled wheels for specific Python versions (3.10–3.13) and Linux platforms (aarch64, x86_64).
- Active maintenance with recent commits and stable production status; last release 20 days old.
License · maintenance · safety
permissive license (permissive) — Permissive license allows commercial and private use with minimal restrictions, typical for infrastructure projects in the AI serving ecosystem.
last release 2026-07-25 (20 days) · last repo commit 2026-08-14 · 6,276 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 270,986 downloads/mo, #8,225 on PyPI
Alternatives
Verify before relying
pip install mooncake-transfer-engine
import mooncake_transfer_engine
# Use with vLLM, SGLang, or other LLM serving frameworks via their Mooncake connectors- Specific RDMA hardware requirements and whether fallback to TCP is supported
- Performance overhead or latency characteristics for different transfer sizes
- Compatibility matrix with specific vLLM, SGLang, and TensorRT LLM versions
What it is and what it does
Mooncake Transfer Engine is a low-level data transfer component for disaggregated LLM serving architectures. It enables zero-copy, high-throughput movement of KV cache and model weights across distributed GPU clusters using RDMA and P2P protocols. The engine sits at the core of Mooncake, a platform that separates prefill and decode compute into independent clusters while pooling KV cache across otherwise idle CPU, DRAM, and SSD resources.
The package is typically not used directly but integrated into higher-level serving frameworks—vLLM, SGLang, TensorRT LLM, LMDeploy, and others have built connectors and backends around it. Its runtime dependencies are aiohttp, requests, and msgpack, suggesting async HTTP communication and serialization for remote cache coordination. It is production-grade software powering Kimi, a leading LLM service, and has been deployed at scale on hundreds of GPUs.
Use it for
- Disaggregated prefill-decode inference: separate prefill and decode clusters while sharing KV cache via Transfer Engine
- Large-scale model training: distribute model weights across thousands of GPUs with fast P2P updates (e.g., 1T-parameter models in ~20 seconds)
- Multimodal embedding caching: cross-instance sharing of Vision Transformer embeddings to avoid redundant GPU computation
- Hierarchical KV cache storage: offload cache to host DRAM or remote storage with intelligent prefetch and eviction
- Encoder-decoder decoupling: separate compute-intensive encoders from language model nodes with zero-copy embedding transfer
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you are building or deploying a disaggregated LLM inference or training system at scale and your framework (vLLM, SGLang, TensorRT LLM, LMDeploy) has native Mooncake support.
Do not install directly for single-machine use; it is an infrastructure component for multi-node GPU clusters. Active maintenance, no known vulnerabilities, and permissive license make it safe to adopt in production.
Install
mooncake-transfer-engine on PyPI
Before you install
Medium install friction due to compiled wheels for specific Python versions (3.10–3.13) and Linux platforms (aarch64, x86_64). Active maintenance with recent commits and stable production status; last release 20 days old.
Requires Linux (manylinux_2_28), Python 3.10 or later, and NVIDIA CUDA environment; designed for GPU cluster deployment, not single-machine use.
License in practice
Permissive license allows commercial and private use with minimal restrictions, typical for infrastructure projects in the AI serving ecosystem.
Quickstart
pip install mooncake-transfer-engine
import mooncake_transfer_engine
# Use with vLLM, SGLang, or other LLM serving frameworks via their Mooncake connectorsVerify before relying
- Specific RDMA hardware requirements and whether fallback to TCP is supported
- Performance overhead or latency characteristics for different transfer sizes
- Compatibility matrix with specific vLLM, SGLang, and TensorRT LLM versions
Package facts
| License | permissive license permissive |
| Python support | Supports the current Python release >=3.10 |
| Install friction | Medium. Platform-specific wheel |
| Runtime dependencies | 3 packagesaiohttprequestsmsgpack |
| Maintenance | Actively maintained 20 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 270,986 / month, #8,225 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 5 - Production/StableEnvironment :: GPU :: NVIDIA CUDAIntended Audience :: DevelopersIntended Audience :: Science/ResearchLicense :: OSI Approved :: Apache Software LicenseOperating System :: POSIX :: LinuxProgramming Language :: C++Programming Language :: Python :: 3Programming Language :: Python :: 3 :: OnlyProgramming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: Implementation :: CPythonTopic :: Scientific/Engineering :: Artificial IntelligenceTopic :: System :: Distributed Computing |
Evidence: mooncake_transfer_engine-0.3.12.post1-cp310-cp310-manylinux_2_28_aarch64.whl; mooncake_transfer_engine-0.3.12.post1-cp310-cp310-manylinux_2_28_x86_64.whl; mooncake_transfer_engine-0.3.12.post1-cp311-cp311-manylinux_2_28_aarch64.whl; mooncake_transfer_engine-0.3.12.post1-cp311-cp311-manylinux_2_28_x86_64.whl; mooncake_transfer_engine-0.3.12.post1-cp312-cp312-manylinux_2_28_aarch64.whl; mooncake_transfer_engine-0.3.12.post1-cp312-cp312-manylinux_2_28_x86_64.whl; mooncake_transfer_engine-0.3.12.post1-cp313-cp313-manylinux_2_28_aarch64.whl; mooncake_transfer_engine-0.3.12.post1-cp313-cp313-manylinux_2_28_x86_64.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “rdma distributed inference”
- mooncake-transfer-engineMooncake Transfer Engine provides high-performance, zero-copy data…
- mooncake-transfer-engine-cuda13Mooncake Transfer Engine provides high-performance, zero-copy…
- memcache-hybridProvides Python bindings for a high-performance distributed key-value…
Give your agent the search over MCP, or paste the wish link into any chat.
More Artificial Intelligence packages
LiteLLM provides a unified Python interface to call 100+ LLM providers (OpenAI, Anthropic, Gemini, Bedrock, Azure, and others) using OpenAI-compatible API format, available as both a Python SDK and a self-hosted AI Gateway proxy server.
Install it if you need to work with multiple LLM providers or want to centralize LLM routing in your organization.
Client library and CLI tool for downloading, uploading, and managing models, datasets, and repositories on the Hugging Face Hub platform.
Install it if you work with Hugging Face Hub models or datasets.
LangChain provides a framework for building agents and LLM-powered applications by composing language models, tools, and memory through a unified API that abstracts over multiple model providers.
hf-xet provides chunk-based deduplication and efficient file transfer for the Hugging Face Hub, enabling faster uploads and downloads of large files with local disk caching.
Tokenizers converts raw text into token sequences for NLP models, with support for training custom vocabularies and using pre-built tokenizers (BPE, WordPiece) optimized for speed via Rust.
Transformers provides a unified framework for loading, fine-tuning, and running state-of-the-art pretrained models across text, vision, audio, video, and multimodal tasks using PyTorch, JAX, or TensorFlow.
Install it if you need to run or train any transformer-based model for NLP, vision, audio, or multimodal tasks.
See also lmcache · memcache-hybrid · mooncake-transfer-engine-cuda13 · vllm-router · vllm · vllm-tpu · sglang-router · vllm-cpu · vineyard · vineyard-bdist