vllm-router

What it is and what it does

vllm-router is a Rust-based request forwarding system designed to distribute inference workloads across multiple vLLM worker instances in large-scale deployments. It sits between clients and a pool of vLLM servers, accepting requests and routing them according to configurable load-balancing policies—round-robin, random, consistent hashing, power-of-two, or cache-aware selection. The router is particularly useful when you need session affinity (routing the same user to the same worker to reuse KV cache) or when you want to optimize for cache hits on repeated prompts.

Beyond basic routing, vllm-router handles prefill-decode disaggregation, where inference is split into two phases: prefill (processing the prompt) and decode (generating tokens). It can route these phases to separate worker pools, allowing independent scaling of each. It includes enterprise-grade features like circuit breakers (to protect workers from cascading failures), automatic retries with exponential backoff, Kubernetes-native worker discovery, bearer-token authentication, and Prometheus metrics collection. The package ships as a compiled binary (built from Rust) with a Python CLI wrapper, so installation requires prebuilt wheels or a Rust toolchain.

Use it for

• Multi-turn chat applications where you want to route the same user to the same worker to preserve KV cache across turns.
• Deployments with repeated prompts or few-shot examples where cache-aware routing maximizes prefix cache reuse.
• Large-scale vLLM clusters running on Kubernetes where automatic worker discovery and health monitoring reduce operational overhead.
• Prefill-decode disaggregated inference where you scale prefill and decode workers independently based on workload demand.
• Production systems requiring circuit breakers and retry logic to gracefully handle worker failures and transient errors.

Worth the install?

AI-flagged interpretation of the facts on this page. Verify before relying on it.

Install

vllm-router on PyPI

Before you install

Medium install friction due to compiled Rust components (wheels provided for x86_64 and aarch64 Linux). Active maintenance with recent releases. Six runtime dependencies (fastapi, uvicorn, aiohttp, requests, orjson, setproctitle) are all stable, widely-used libraries.

Requires Rust and Cargo installed to build from source; prebuilt wheels available for Python 3.8+ on Linux x86_64 and aarch64.

License in practice

Apache-2.0 permissive license allows commercial use, modification, and distribution with minimal restrictions—suitable for enterprise deployments.

Quickstart

pip install vllm-router

# Launch router with consistent hashing load balancing
vllm-router \
  --worker-urls http://worker1:8000 http://worker2:8000 \
  --policy consistent_hash \
  --intra-node-data-parallel-size 8

Package facts

Evidence: vllm_router-0.1.15-cp38-abi3-manylinux_2_28_aarch64.whl; vllm_router-0.1.15-cp38-abi3-manylinux_2_28_x86_64.whl

Tags

More Distributed Computing packages

See also sglang-router · vllm-sr · vllm · mooncake-transfer-engine · mooncake-transfer-engine-cuda13 · vllm-tpu · roundrobin · lmcache · smg-grpc-servicer · semantic-router

Further reading