{"categories":[{"label":"Distributed Computing","url":"https://skillfed.io/packages/category/system-distributed-computing/3"}],"enrichment":{"capability":"Routes and load-balances requests across vLLM worker instances with support for multiple algorithms, prefill-decode disaggregation, and Kubernetes service discovery.","skillfed_tags":["llm-inference","load-balancing","kubernetes"],"use_cases":["Multi-turn chat applications where you want to route the same user to the same worker to preserve KV cache across turns.","Deployments with repeated prompts or few-shot examples where cache-aware routing maximizes prefix cache reuse.","Large-scale vLLM clusters running on Kubernetes where automatic worker discovery and health monitoring reduce operational overhead.","Prefill-decode disaggregated inference where you scale prefill and decode workers independently based on workload demand.","Production systems requiring circuit breakers and retry logic to gracefully handle worker failures and transient errors."],"what_it_does":"vllm-router is a Rust-based request forwarding system designed to distribute inference workloads across multiple vLLM worker instances in large-scale deployments. It sits between clients and a pool of vLLM servers, accepting requests and routing them according to configurable load-balancing policies\u2014round-robin, random, consistent hashing, power-of-two, or cache-aware selection. The router is particularly useful when you need session affinity (routing the same user to the same worker to reuse KV cache) or when you want to optimize for cache hits on repeated prompts.\n\nBeyond basic routing, vllm-router handles prefill-decode disaggregation, where inference is split into two phases: prefill (processing the prompt) and decode (generating tokens). It can route these phases to separate worker pools, allowing independent scaling of each. It includes enterprise-grade features like circuit breakers (to protect workers from cascading failures), automatic retries with exponential backoff, Kubernetes-native worker discovery, bearer-token authentication, and Prometheus metrics collection. The package ships as a compiled binary (built from Rust) with a Python CLI wrapper, so installation requires prebuilt wheels or a Rust toolchain.","worth_installing":"Yes, if you are running vLLM at scale and need intelligent request routing with session affinity or prefill-decode separation. The Apache-2.0 license is permissive, maintenance is active, and there are no known vulnerabilities. Install friction is moderate due to compiled Rust components, but prebuilt wheels are available. Not necessary for single-worker or simple round-robin deployments."},"id":"vllm-router","links":{"html":"https://skillfed.io/packages/vllm-router","md":"https://skillfed.io/packages/vllm-router.md","pypi":"https://pypi.org/project/vllm-router/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-07-13","license_spdx":"Apache-2.0","license_treatment":"permissive","name":"vllm-router","python_support":"supports_current","summary":"High-performance Rust-based load balancer for VLLM with multiple routing algorithms and prefill-decode disaggregation support"},"popularity":{"monthly_downloads":74317,"position":14851,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.1.15"}