{"categories":[{"label":"Distributed Computing","url":"https://skillfed.io/packages/category/system-distributed-computing"}],"enrichment":{"capability":"A high-performance Rust-based load balancer and gateway for routing requests across distributed SGLang model workers, supporting multiple load-balancing strategies, prefill-decode disaggregation, and OpenAI-compatible APIs.","skillfed_tags":["inference-gateway","load-balancing","llm-serving"],"use_cases":["Route inference requests across multiple SGLang workers with cache-aware load balancing to minimize redundant computation.","Separate prefill and decode stages across dedicated worker pools to optimize throughput and latency independently.","Proxy OpenAI API requests to multiple remote vendors while maintaining streaming semantics and conversation state.","Monitor and control multi-model inference deployments with per-model policies, circuit breakers, and dynamic worker discovery.","Serve reasoning-capable models with native tokenization and tool-call parsing via gRPC.","Centralize conversation history and state management across model fleets with pluggable storage backends."],"what_it_does":"sglang-router is a Rust-based gateway that orchestrates distributed inference workloads across fleets of SGLang model workers. It sits between clients and worker nodes, routing requests using pluggable strategies (random, round_robin, cache_aware, power_of_two, bucket), managing prefill-decode disaggregation for efficient batching, and exposing OpenAI-compatible REST endpoints. The gateway handles reliability concerns\u2014retries with backoff, circuit breakers, token-bucket rate limiting, and request queuing\u2014while maintaining observability through Prometheus metrics and OpenTelemetry tracing.\n\nThe package is designed for large-scale LLM deployments where traffic needs to be balanced across heterogeneous worker pools, models need to be served concurrently, and inference stages (prefill and decode) can be separated for independent scaling. It integrates with SGLang's serving runtime deeply, including native Rust tokenization, reasoning parsing, and tool-call execution, while also supporting external OpenAI-compatible backends. Runtime dependencies (fastapi, uvicorn, aiohttp, orjson, setproctitle) enable the Python launcher and async HTTP handling.","worth_installing":"Yes, if you are deploying distributed SGLang inference at scale and need sophisticated load balancing, prefill-decode disaggregation, or multi-model orchestration. The package is in the top 15000 on PyPI, has no known vulnerabilities, and offers production-grade observability and reliability features. The aging maintenance status (211 days since last release) is a minor concern for a specialized infrastructure tool but warrants monitoring for security updates. Not recommended for simple single-worker or non-SGLang deployments."},"id":"sglang-router","links":{"html":"https://skillfed.io/packages/sglang-router","md":"https://skillfed.io/packages/sglang-router.md","pypi":"https://pypi.org/project/sglang-router/"},"maintenance":{"status":"aging"},"meta":{"latest_release":"2026-01-15","license_spdx":null,"license_treatment":"permissive","name":"sglang-router","python_support":"supports_current","summary":"High-performance Rust-based load balancer for SGLang with multiple routing algorithms and prefill-decode disaggregation support"},"popularity":{"monthly_downloads":680213,"position":5366,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.3.2"}