multi-tenant-llm-hosting
Build shared LLM inference platforms that safely serve multiple teams or customers without compromising security or performance. This skill covers tenant identity enforcement, per-tenant API keys, namespace isolation, quota configuration, request routing, and cost attribution—enabling you to run efficient, governed multi-tenant deployments on Kubernetes with vLLM and API gateways.
Multi-Tenant LLM Hosting helps you run shared inference infrastructure for multiple teams or customers with tenant isolation, quotas, and cost controls.
AI-generated summary based on this skill's SKILL.md
Install
BagelHole/DevOps-Security-Agent-Skills/multi-tenant-llm-hosting · repository language: Shell
git clone https://github.com/BagelHole/DevOps-Security-Agent-Skills
cp -r DevOps-Security-Agent-Skills/infrastructure/local-ai/multi-tenant-llm-hosting ~/.claude/skills/multi-tenant-llm-hostingnpx skillfed install BagelHole/DevOps-Security-Agent-Skills/multi-tenant-llm-hostingFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do you isolate tenants on shared GPU infrastructure?
multi-tenant-llm-hosting isolates tenants through Kubernetes namespaces, network policies, and RBAC rules that prevent cross-tenant access. GPU quotas are enforced per namespace, and request routing uses per-tenant API keys to ensure each customer's workload stays logically and physically separated. vLLM model instances can be deployed in dedicated namespace pods with resource limits, blocking noisy-neighbor interference.
What does multi-tenant-llm-hosting enable for cost attribution?
multi-tenant-llm-hosting integrates billing and usage tracking by tagging every inference request with a tenant identifier, then aggregating token counts, GPU time, and API calls per customer. Cost attribution flows through API gateway logs and Kubernetes metrics into a billing system, allowing you to recover infrastructure costs and charge customers based on actual consumption rather than flat fees.
How can you implement per-tenant quotas and rate limiting?
multi-tenant-llm-hosting applies per-tenant quotas via Kubernetes ResourceQuotas and LimitRanges in each namespace, plus API gateway rate-limiting rules keyed to tenant API keys. Request routers check quotas before forwarding to vLLM backends, rejecting excess requests with 429 responses. This prevents any single tenant from exhausting shared GPU or memory resources.
What security measures prevent cross-tenant data leakage?
multi-tenant-llm-hosting uses namespace isolation, network policies blocking inter-namespace traffic, and RBAC to restrict service account permissions. API keys authenticate tenant identity before request routing, and model cache/KV storage is partitioned per tenant. Kubernetes audit logs track all access, enabling compliance verification and incident response.
How does multi-tenant-llm-hosting prevent noisy-neighbor issues?
multi-tenant-llm-hosting prevents noisy-neighbor problems by enforcing per-tenant GPU and memory quotas, prioritizing requests with QoS classes, and isolating model instances in separate pods. Fair resource sharing is maintained through Kubernetes scheduling constraints and vLLM queue management, ensuring one tenant's spike doesn't starve others.
What is the typical architecture for multi-tenant LLM platform design?
multi-tenant-llm-hosting follows a layered architecture: API gateway (tenant authentication and rate limiting), request router (quota checks and tenant tagging), vLLM inference pods (namespace-isolated), and billing aggregator (usage tracking). Each tenant gets a Kubernetes namespace with ResourceQuotas, and metrics flow to a central billing system for cost recovery and reporting.
SKILL.md
rendered from the published skill — quoted content, verbatim
Multi-Tenant LLM Hosting
Host many teams/customers on shared inference infrastructure without sacrificing security, performance, or cost governance.
When to Use This Skill
- Building an internal LLM platform shared by multiple teams
- Hosting LLM inference for external customers with isolation requirements
- Implementing per-tenant quotas, billing, and rate limiting
- Designing request routing for multi-model, multi-tenant environments
- Preventing noisy-neighbor issues on shared GPU infrastructure
Prerequisites
- Kubernetes cluster with GPU node pools
- API gateway or LLM gateway (LiteLLM, Envoy, Kong)
- Prometheus + Grafana for per-tenant observability
- Redis or equivalent for rate limiting state
- Billing system or cost attribution database
Isolation Model
- Strong tenant identity on every request
- Per-tenant API keys and scoped model access
- Namespace or workload isolation for
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 1 file
infrastructure/local-ai/multi-tenant-llm-hosting/SKILL.md