vllm-router

What it is and what it does

vllm-router is a Rust-based request forwarding system designed to distribute inference workloads across multiple vLLM worker instances in large-scale deployments. It sits between clients and a pool of vLLM servers, accepting requests and routing them according to configurable load-balancing policies—round-robin, random, consistent hashing, power-of-two, or cache-aware selection. The router is particularly useful when you need session affinity (routing the same user to the same worker to reuse KV cache) or when you want to optimize for cache hits on repeated prompts.

Beyond basic routing, vllm-router handles prefill-decode disaggregation, where inference is split into two phases: prefill (processing the prompt) and decode (generating tokens). It can route these phases to separate worker pools, allowing independent scaling of each. It includes enterprise-grade features like circuit breakers (to protect workers from cascading failures), automatic retries with exponential backoff, Kubernetes-native worker discovery, bearer-token authentication, and Prometheus metrics collection. The package ships as a compiled binary (built from Rust) with a Python CLI wrapper, so installation requires prebuilt wheels or a Rust toolchain.

Use it for:

• Multi-turn chat applications where you want to route the same user to the same worker to preserve KV cache across turns.
• Deployments with repeated prompts or few-shot examples where cache-aware routing maximizes prefix cache reuse.
• Large-scale vLLM clusters running on Kubernetes where automatic worker discovery and health monitoring reduce operational overhead.
• Prefill-decode disaggregated inference where you scale prefill and decode workers independently based on workload demand.
• Production systems requiring circuit breakers and retry logic to gracefully handle worker failures and transient errors.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Routes and load-balances requests across vLLM worker instances with support for multiple algorithms, prefill-decode disaggregation, and Kubernetes service discovery.

Yes, if you are running vLLM at scale and need intelligent request routing with session affinity or prefill-decode separation. The Apache-2.0 license is permissive, maintenance is active, and there are no known vulnerabilities. Install friction is moderate due to compiled Rust components, but prebuilt wheels are available. Not necessary for single-worker or simple round-robin deployments.

Install

vllm-router on PyPI

pip install vllm-router

uv add vllm-router

poetry add vllm-router

Installing vllm-router

Before you install

Medium install friction due to compiled Rust components (wheels provided for x86_64 and aarch64 Linux). Active maintenance with recent releases. Six runtime dependencies (fastapi, uvicorn, aiohttp, requests, orjson, setproctitle) are all stable, widely-used libraries.

License in practice

Apache-2.0 permissive license allows commercial use, modification, and distribution with minimal restrictions—suitable for enterprise deployments.

Quickstart

pip install vllm-router

# Launch router with consistent hashing load balancing
vllm-router \
  --worker-urls http://worker1:8000 http://worker2:8000 \
  --policy consistent_hash \
  --intra-node-data-parallel-size 8

Requires Rust and Cargo installed to build from source; prebuilt wheels available for Python 3.8+ on Linux x86_64 and aarch64.

Verify before relying

• Performance characteristics (latency, throughput) compared to alternatives under typical vLLM deployment scales.
• Compatibility with specific vLLM versions and connector types (NCCL, Mooncake, NIXL).
• Operational overhead of Kubernetes service discovery in large clusters.

Package facts

Evidence: vllm_router-0.1.15-cp38-abi3-manylinux_2_28_aarch64.whl; vllm_router-0.1.15-cp38-abi3-manylinux_2_28_x86_64.whl

Tags

More Distributed Computing packages

Further reading