sglang-router

What it is and what it does

sglang-router is a Rust-based gateway that orchestrates distributed inference workloads across fleets of SGLang model workers. It sits between clients and worker nodes, routing requests using pluggable strategies (random, round_robin, cache_aware, power_of_two, bucket), managing prefill-decode disaggregation for efficient batching, and exposing OpenAI-compatible REST endpoints. The gateway handles reliability concerns—retries with backoff, circuit breakers, token-bucket rate limiting, and request queuing—while maintaining observability through Prometheus metrics and OpenTelemetry tracing.

The package is designed for large-scale LLM deployments where traffic needs to be balanced across heterogeneous worker pools, models need to be served concurrently, and inference stages (prefill and decode) can be separated for independent scaling. It integrates with SGLang's serving runtime deeply, including native Rust tokenization, reasoning parsing, and tool-call execution, while also supporting external OpenAI-compatible backends. Runtime dependencies (fastapi, uvicorn, aiohttp, orjson, setproctitle) enable the Python launcher and async HTTP handling.

Use it for

• Route inference requests across multiple SGLang workers with cache-aware load balancing to minimize redundant computation.
• Separate prefill and decode stages across dedicated worker pools to optimize throughput and latency independently.
• Proxy OpenAI API requests to multiple remote vendors while maintaining streaming semantics and conversation state.
• Monitor and control multi-model inference deployments with per-model policies, circuit breakers, and dynamic worker discovery.
• Serve reasoning-capable models with native tokenization and tool-call parsing via gRPC.
• Centralize conversation history and state management across model fleets with pluggable storage backends.

Worth the install?

AI-flagged interpretation of the facts on this page. Verify before relying on it.

Install

sglang-router on PyPI

Before you install

Medium install friction due to Rust compilation requirements and system dependencies (libssl-dev or openssl-devel). The package provides pre-built wheels for Python 3.8+ across major platforms (x86_64, ARM64, macOS, Linux, Windows), reducing friction for standard environments. Maintenance status is aging—last release was 211 days ago—so expect slower updates.

Requires Rust toolchain (rustc, cargo) if building from source; system OpenSSL development headers (libssl-dev or openssl-devel) for development builds. Pre-built wheels bypass Rust compilation for standard platforms.

License in practice

Licensed under Apache-2.0 (permissive), allowing commercial use, modification, and redistribution with minimal restrictions. No notable licensing constraints for most use cases.

Quickstart

pip install sglang-router

python3 -m sglang_router.launch_router \
  --worker-urls http://worker1:8000 http://worker2:8000 \
  --policy cache_aware

Package facts

Evidence: sglang_router-0.3.2-cp38-abi3-macosx_10_12_x86_64.whl; sglang_router-0.3.2-cp38-abi3-macosx_11_0_arm64.whl; sglang_router-0.3.2-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; sglang_router-0.3.2-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; sglang_router-0.3.2-cp38-abi3-musllinux_1_1_aarch64.whl; sglang_router-0.3.2-cp38-abi3-musllinux_1_1_x86_64.whl; sglang_router-0.3.2-cp38-abi3-win_amd64.whl

Tags

More Distributed Computing packages

See also vllm-router · sanic-routing · sglang · vllm · vllm-sr · mooncake-transfer-engine · vllm-tpu · smg-grpc-servicer · semantic-router · pybgpkit-parser