---
id: mooncake-transfer-engine-cuda13
version: "0.3.12.post1"
license: unclear
license_treatment: permissive
maintenance: active
---

# mooncake-transfer-engine-cuda13 — A KVCache-centric Disaggregated Architecture for large-scale LLM inference and training. (CUDA 13 version)

License: permissive · Maintenance: active · Downloads: 158.0K/mo

## What it is and what it does

Mooncake Transfer Engine is a component of the Mooncake infrastructure platform designed for large-scale LLM inference and training. It implements a KV cache-centric disaggregated architecture that separates prefill and decode compute clusters while using underutilized CPU, DRAM, and SSD resources to build a distributed KV cache pool. The engine provides zero-copy RDMA-based transfer of KV cache data across GPU clusters, enabling efficient cross-instance sharing of cached key-value states.

The package is tightly integrated with major LLM serving frameworks including vLLM, SGLang, TensorRT LLM, and LMDeploy. It serves as a backend for distributed KV cache management, hierarchical caching across device/host/remote storage tiers, and disaggregated prefill-decode inference patterns. The transfer engine handles large-scale model updates and multimodal embedding distribution in production training and inference pipelines.

Use it for:

- Disaggregated LLM inference on multi-node GPU clusters to separate compute-intensive prefill from decode operations
- Distributed KV cache pooling to reduce per-node memory pressure and enable higher request throughput
- Zero-copy cross-GPU weight and embedding transfer during large-scale distributed training
- Hierarchical KV cache storage with intelligent offloading across device, host, and remote tiers
- Multimodal inference with efficient cross-instance sharing of encoder outputs (e.g., Vision Transformer embeddings)

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Mooncake Transfer Engine provides high-performance, zero-copy RDMA-based data transfer for distributed KV cache management in large-scale LLM inference and training systems.

Yes, if you are deploying large-scale LLM inference or training on multi-node GPU clusters and need efficient distributed KV cache management. The package is production-stable, actively maintained, and integrates with established frameworks (vLLM, SGLang, TensorRT LLM). Requires CUDA 13 and Python 3.10+; not suitable for single-node or CPU-only setups.

## Install

pip install mooncake-transfer-engine-cuda13
uv add mooncake-transfer-engine-cuda13
poetry add mooncake-transfer-engine-cuda13

## Installing mooncake-transfer-engine-cuda13

Before you install: Medium install friction due to CUDA 13 requirement and platform-specific wheels (manylinux_2_28 for x86_64 and aarch64). Package is actively maintained with recent commits and production-stable status. Requires Python 3.10+.

License in practice: Permissive license treatment allows commercial and private use without significant restrictions.

Quickstart:

pip install mooncake-transfer-engine-cuda13

import mooncake_transfer_engine_cuda13
# Use with vLLM or SGLang via their Mooncake connectors
# or directly via the Transfer Engine API for custom RDMA transfers

Requires CUDA 13 runtime, NVIDIA GPU with RDMA support, and manylinux_2_28 compatible Linux system (x86_64 or aarch64). Python 3.10 or later.

Verify before relying:

- Whether CUDA 13 runtime is required on the system or if the package bundles it
- Specific performance gains or throughput numbers for typical workloads
- Compatibility with non-NVIDIA GPU architectures or inference frameworks beyond those mentioned

## Package facts

- License: not declared (permissive)
- Python support: supports_current
- Install friction: medium
- Maintenance: active
- Downloads: 158.0K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

llm kv cache transfer, rdma distributed inference, disaggregated llm serving, gpu cluster data transfer, kv cache pool management, llm-inference, distributed-systems, gpu-acceleration

[View on SkillFed](https://skillfed.io/packages/mooncake-transfer-engine-cuda13) · [View on PyPI](https://pypi.org/project/mooncake-transfer-engine-cuda13/)