sglang-router

What it is and what it does

sglang-router is a Rust-based gateway that orchestrates distributed inference workloads across fleets of SGLang model workers. It sits between clients and worker nodes, routing requests using pluggable strategies (random, round_robin, cache_aware, power_of_two, bucket), managing prefill-decode disaggregation for efficient batching, and exposing OpenAI-compatible REST endpoints. The gateway handles reliability concerns—retries with backoff, circuit breakers, token-bucket rate limiting, and request queuing—while maintaining observability through Prometheus metrics and OpenTelemetry tracing.

The package is designed for large-scale LLM deployments where traffic needs to be balanced across heterogeneous worker pools, models need to be served concurrently, and inference stages (prefill and decode) can be separated for independent scaling. It integrates with SGLang's serving runtime deeply, including native Rust tokenization, reasoning parsing, and tool-call execution, while also supporting external OpenAI-compatible backends. Runtime dependencies (fastapi, uvicorn, aiohttp, orjson, setproctitle) enable the Python launcher and async HTTP handling.

Use it for:

• Route inference requests across multiple SGLang workers with cache-aware load balancing to minimize redundant computation.
• Separate prefill and decode stages across dedicated worker pools to optimize throughput and latency independently.
• Proxy OpenAI API requests to multiple remote vendors while maintaining streaming semantics and conversation state.
• Monitor and control multi-model inference deployments with per-model policies, circuit breakers, and dynamic worker discovery.
• Serve reasoning-capable models with native tokenization and tool-call parsing via gRPC.
• Centralize conversation history and state management across model fleets with pluggable storage backends.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

A high-performance Rust-based load balancer and gateway for routing requests across distributed SGLang model workers, supporting multiple load-balancing strategies, prefill-decode disaggregation, and OpenAI-compatible APIs.

Yes, if you are deploying distributed SGLang inference at scale and need sophisticated load balancing, prefill-decode disaggregation, or multi-model orchestration. The package is in the top 15000 on PyPI, has no known vulnerabilities, and offers production-grade observability and reliability features. The aging maintenance status (211 days since last release) is a minor concern for a specialized infrastructure tool but warrants monitoring for security updates. Not recommended for simple single-worker or non-SGLang deployments.

Install

sglang-router on PyPI

pip install sglang-router

uv add sglang-router

poetry add sglang-router

Installing sglang-router

Before you install

Medium install friction due to Rust compilation requirements and system dependencies (libssl-dev or openssl-devel). The package provides pre-built wheels for Python 3.8+ across major platforms (x86_64, ARM64, macOS, Linux, Windows), reducing friction for standard environments. Maintenance status is aging—last release was 211 days ago—so expect slower updates.

License in practice

Licensed under Apache-2.0 (permissive), allowing commercial use, modification, and redistribution with minimal restrictions. No notable licensing constraints for most use cases.

Quickstart

pip install sglang-router

python3 -m sglang_router.launch_router \
  --worker-urls http://worker1:8000 http://worker2:8000 \
  --policy cache_aware

Requires Rust toolchain (rustc, cargo) if building from source; system OpenSSL development headers (libssl-dev or openssl-devel) for development builds. Pre-built wheels bypass Rust compilation for standard platforms.

Verify before relying

• Performance benchmarks comparing different load-balancing strategies under realistic LLM workloads.
• Kubernetes integration maturity and production deployment patterns.
• Compatibility with non-SGLang model serving backends beyond OpenAI-compatible proxying.
• Scaling limits and performance characteristics at very large worker fleet sizes.

Package facts

Evidence: sglang_router-0.3.2-cp38-abi3-macosx_10_12_x86_64.whl; sglang_router-0.3.2-cp38-abi3-macosx_11_0_arm64.whl; sglang_router-0.3.2-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; sglang_router-0.3.2-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; sglang_router-0.3.2-cp38-abi3-musllinux_1_1_aarch64.whl; sglang_router-0.3.2-cp38-abi3-musllinux_1_1_x86_64.whl; sglang_router-0.3.2-cp38-abi3-win_amd64.whl

Tags

More Distributed Computing packages