llm-inference-scaling
This skill enables dynamic scaling of LLM inference workloads across Kubernetes clusters using KEDA and Prometheus metrics tied to GPU utilization and request queues. It covers vLLM deployment, queue-based job scaling with Redis, spot instance strategies, and cluster autoscaler configuration to handle traffic spikes while optimizing costs.
LLM Inference Scaling auto-scales LLM inference pods on Kubernetes using KEDA, GPU metrics, and spot instances.
AI-generated summary based on this skill's SKILL.md
Install
BagelHole/DevOps-Security-Agent-Skills/llm-inference-scaling · repository language: Shell
git clone https://github.com/BagelHole/DevOps-Security-Agent-Skills
cp -r DevOps-Security-Agent-Skills/infrastructure/local-ai/llm-inference-scaling ~/.claude/skills/llm-inference-scalingnpx skillfed install BagelHole/DevOps-Security-Agent-Skills/llm-inference-scalingFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How does llm-inference-scaling enable auto-scale LLM pods on Kubernetes?
llm-inference-scaling provides KEDA-based autoscaling that monitors GPU utilization and request queue depth to dynamically scale LLM inference pods. It integrates Prometheus metrics collection with vLLM deployments, allowing the system to add or remove pod replicas based on real-time GPU load and incoming API traffic patterns, ensuring efficient resource use across your Kubernetes cluster.
What role does KEDA play in GPU-aware scaling with this skill?
llm-inference-scaling uses KEDA to connect custom Prometheus metrics—including GPU utilization and Redis queue length—to Kubernetes' horizontal pod autoscaler. KEDA acts as the bridge between your monitoring stack and scaling decisions, enabling GPU-aware triggers that traditional HPA cannot achieve, so LLM workloads scale intelligently based on actual accelerator demand.
Can llm-inference-scaling reduce LLM serving costs using spot instances?
Yes. llm-inference-scaling includes strategies for deploying LLM inference on spot instances and preemptible GPUs within Kubernetes, combined with cluster autoscaler configuration. By pairing queue-based scaling with cost-aware node provisioning, it minimizes expenses during traffic lulls while maintaining performance during spikes, reducing overall LLM serving infrastructure costs.
How does llm-inference-scaling handle unpredictable LLM API traffic?
llm-inference-scaling implements queue-based autoscaling using Redis or similar systems to buffer incoming requests. KEDA monitors queue depth via Prometheus and triggers pod scaling before requests timeout. This decouples request arrival from processing capacity, allowing the system to absorb traffic spikes gracefully and scale compute resources to match queue buildup.
What metrics does llm-inference-scaling use for scaling decisions?
llm-inference-scaling relies on Prometheus metrics including GPU utilization, GPU memory usage, request queue length (from Redis), and vLLM-specific counters like pending requests. These metrics feed KEDA scalers, enabling multi-dimensional scaling logic that reacts to both resource saturation and workload backlog, not just CPU or memory.
Does llm-inference-scaling support managing multiple LLM models?
llm-inference-scaling can orchestrate multi-model LLM serving by scaling separate deployment groups per model while sharing GPU node pools. Each model's queue and GPU metrics are tracked independently in Prometheus, allowing KEDA to balance scaling decisions across models and maximize shared GPU resource utilization on your Kubernetes cluster.
SKILL.md
rendered from the published skill — quoted content, verbatim
LLM Inference Scaling
Scale LLM inference horizontally on Kubernetes with GPU-aware autoscaling, request queuing, and cost-efficient spot instance strategies.
When to Use This Skill
Use this skill when: - LLM API traffic is unpredictable and you need to scale up/down automatically - Managing a fleet of vLLM or TGI inference pods on Kubernetes - Reducing inference costs with spot/preemptible GPU instances - Implementing queue-based autoscaling for batch inference jobs - Building a multi-model serving platform that shares GPU resources
Prerequisites
- Kubernetes cluster with GPU nodes (NVIDIA operator installed)
- KEDA (Kubernetes Event-Driven Autoscaler) installed
- Prometheus with GPU metrics (
dcgm-exporterorgpu-operator) - Helm 3+ for chart deployments
GPU Node Setup
```bash
Install NVIDIA GPU Operator (handles drivers,
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 1 file
infrastructure/local-ai/llm-inference-scaling/SKILL.md