llm-inference-scaling
This skill enables dynamic scaling of LLM inference workloads across Kubernetes clusters using KEDA and Prometheus metrics tied to GPU utilization and request queues. It covers vLLM deployment, queue-based job scaling with Redis, spot instance strategies, and cluster autoscaler configuration to handle traffic spikes while optimizing costs.
LLM Inference Scaling auto-scales LLM inference pods on Kubernetes using KEDA, GPU metrics, and spot instances.
AI-generated summary based on this skill's SKILL.md
Decision gist · record as of 2026-05-22
LLM Inference Scaling auto-scales LLM inference pods on Kubernetes using KEDA, GPU metrics, and spot instances. This skill enables dynamic scaling of LLM inference workloads across Kubernetes clusters using KEDA and Prometheus metrics tied to GPU utilization and request queues. It covers vLLM deployment, queue-based job scaling with Redis, spot instance strategies, and cluster autoscaler configuration to handle traffic spikes while optimizing costs.
Use it when
- llm-inference-scaling uses KEDA to connect custom Prometheus metrics—including GPU utilization and Redis queue length—to Kubernetes'.
- Yes.
Verify before relying
Read SKILL.md below before installing (1 file). Open directory: indexed for reading, not audited.
Install
BagelHole/DevOps-Security-Agent-Skills/llm-inference-scaling · repository language: Shell
Open directory. Skills are indexed for reading, not audited. Review a skill's body before installing it.
Frequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How does llm-inference-scaling enable auto-scale LLM pods on Kubernetes?
llm-inference-scaling provides KEDA-based autoscaling that monitors GPU utilization and request queue depth to dynamically scale LLM inference pods. It integrates Prometheus metrics collection with vLLM deployments, allowing the system to add or remove pod replicas based on real-time GPU load and incoming API traffic patterns, ensuring efficient resource use across your Kubernetes cluster.
What role does KEDA play in GPU-aware scaling with this skill?
llm-inference-scaling uses KEDA to connect custom Prometheus metrics—including GPU utilization and Redis queue length—to Kubernetes' horizontal pod autoscaler. KEDA acts as the bridge between your monitoring stack and scaling decisions, enabling GPU-aware triggers that traditional HPA cannot achieve, so LLM workloads scale intelligently based on actual accelerator demand.
Can llm-inference-scaling reduce LLM serving costs using spot instances?
Yes. llm-inference-scaling includes strategies for deploying LLM inference on spot instances and preemptible GPUs within Kubernetes, combined with cluster autoscaler configuration. By pairing queue-based scaling with cost-aware node provisioning, it minimizes expenses during traffic lulls while maintaining performance during spikes, reducing overall LLM serving infrastructure costs.
How does llm-inference-scaling handle unpredictable LLM API traffic?
llm-inference-scaling implements queue-based autoscaling using Redis or similar systems to buffer incoming requests. KEDA monitors queue depth via Prometheus and triggers pod scaling before requests timeout. This decouples request arrival from processing capacity, allowing the system to absorb traffic spikes gracefully and scale compute resources to match queue buildup.
What metrics does llm-inference-scaling use for scaling decisions?
llm-inference-scaling relies on Prometheus metrics including GPU utilization, GPU memory usage, request queue length (from Redis), and vLLM-specific counters like pending requests. These metrics feed KEDA scalers, enabling multi-dimensional scaling logic that reacts to both resource saturation and workload backlog, not just CPU or memory.
Does llm-inference-scaling support managing multiple LLM models?
llm-inference-scaling can orchestrate multi-model LLM serving by scaling separate deployment groups per model while sharing GPU node pools. Each model's queue and GPU metrics are tracked independently in Prometheus, allowing KEDA to balance scaling decisions across models and maximize shared GPU resource utilization on your Kubernetes cluster.
SKILL.md
Rendered from the published skill. Quoted content, verbatim.
LLM Inference Scaling
Scale LLM inference horizontally on Kubernetes with GPU-aware autoscaling, request queuing, and cost-efficient spot instance strategies.
When to Use This Skill
Use this skill when: - LLM API traffic is unpredictable and you need to scale up/down automatically - Managing a fleet of vLLM or TGI inference pods on Kubernetes - Reducing inference costs with spot/preemptible GPU instances - Implementing queue-based autoscaling for batch inference jobs - Building a multi-model serving platform that shares GPU resources
Prerequisites
- Kubernetes cluster with GPU nodes (NVIDIA operator installed)
- KEDA (Kubernetes Event-Driven Autoscaler) installed
- Prometheus with GPU metrics (
dcgm-exporterorgpu-operator) - Helm 3+ for chart deployments
GPU Node Setup
```bash
Install NVIDIA GPU Operator (handles drivers,
(truncated - see the full file via the links below)
File tree — 1 file
infrastructure/local-ai/llm-inference-scaling/SKILL.md
Let your AI agent find skills like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 56,283 agent skills by what they can do, searchable in plain language.
wish › “Auto-scale LLM inference workloads on Kubernetes with GPU awareness”
Give your agent the search over MCP, or paste the wish link into any chat. No install? Search from any chat →
Related skills
Deploy and operate production-grade GPU clusters in Kubernetes with built-in support for NVIDIA device plugins, MIG partitioning, and time-slicing. Monitor GPU health via DCGM metrics and Prometheus, configure autoscaling policies, and troubleshoot scheduling and driver issues across your AI infrastructure.
Run production ML inference on Kubernetes using KServe or NVIDIA Triton, with built-in support for canary traffic splitting, request-based autoscaling, and GPU resource allocation. The skill covers model versioning, A/B testing patterns, and dynamic batching for throughput optimization.
Build shared LLM inference platforms that safely serve multiple teams or customers without compromising security or performance. This skill covers tenant identity enforcement, per-tenant API keys, namespace isolation, quota configuration, request routing, and cost attribution—enabling you to run efficient, governed multi-tenant deployments on Kubernetes with vLLM and API gateways.
AKS Automatic 2025 is a fully-managed Kubernetes offering that handles cluster operations, security patching, and node provisioning automatically. It includes Karpenter-based dynamic scaling, Microsoft Entra integration, Azure CNI Overlay networking with Cilium, and built-in monitoring through Azure Monitor. Use this skill to deploy production clusters, configure autoscaling with HPA/VPA/KEDA, set up workload identity, and understand the new billing model.
Model Serving equips you to run LLMs and traditional ML models in production using optimized inference engines like vLLM and BentoML. It covers GPU memory optimization, continuous batching, streaming response patterns, and integration with frontend applications for real-world AI deployments.
Set up automatic scaling for Kubernetes workloads using HPA for CPU/memory-based scaling, VPA for resource optimization, and KEDA for event-driven scenarios like queue processing and scheduled scaling. Includes tools for detecting installations, managing scaled objects, and troubleshooting common scaling issues.
More skills llm-fine-tuning (MIT) · llmops-platform-engineering (MIT) · gpu-server-management (MIT) · Llama Cpp (Apache-2.0)