model-serving-kubernetes
Run production ML inference on Kubernetes using KServe or NVIDIA Triton, with built-in support for canary traffic splitting, request-based autoscaling, and GPU resource allocation. The skill covers model versioning, A/B testing patterns, and dynamic batching for throughput optimization.
Model Serving on Kubernetes deploys ML models at scale using KServe and Triton with canary rollouts and autoscaling.
AI-generated summary based on this skill's SKILL.md
Decision gist · record as of 2026-05-22
Model Serving on Kubernetes deploys ML models at scale using KServe and Triton with canary rollouts and autoscaling. Run production ML inference on Kubernetes using KServe or NVIDIA Triton, with built-in support for canary traffic splitting, request-based autoscaling, and GPU resource allocation. The skill covers model versioning, A/B testing patterns, and dynamic batching for throughput optimization.
Use it when
- Yes.
- model-serving-kubernetes autoscales inference workloads based on GPU and request metrics.
Verify before relying
Read SKILL.md below before installing (1 file). Open directory: indexed for reading, not audited.
Similar skills
Install
BagelHole/DevOps-Security-Agent-Skills/model-serving-kubernetes · repository language: Shell
Open directory. Skills are indexed for reading, not audited. Review a skill's body before installing it.
Frequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I deploy machine learning models on Kubernetes?
model-serving-kubernetes enables production ML inference on Kubernetes using KServe or NVIDIA Triton. You define InferenceService resources that handle model deployment, versioning, and traffic routing. KServe abstracts Kubernetes complexity with built-in model server integration, while Triton offers lower-level control for high-throughput batch inference. Both support GPU allocation and automatic scaling based on request metrics.
Can model-serving-kubernetes implement canary deployments for ML models?
Yes. model-serving-kubernetes supports canary deployments through traffic splitting—route a percentage of requests to new model versions while keeping most traffic on stable versions. This enables safe rollouts and A/B testing of model updates. You configure traffic percentages in the InferenceService spec, allowing gradual validation before full promotion.
What autoscaling options does model-serving-kubernetes provide for GPU workloads?
model-serving-kubernetes autoscales inference workloads based on GPU and request metrics. It integrates with Kubernetes' Horizontal Pod Autoscaler to spawn inference pods when demand rises and shrink when idle. You set target metrics like request latency or GPU utilization, enabling cost-efficient scaling of expensive GPU resources.
How does model-serving-kubernetes handle multiple model versions in production?
model-serving-kubernetes manages multiple model versions through traffic splitting and canary patterns. Deploy different versions as separate predictors within a single InferenceService, then allocate traffic percentages to each. This supports gradual rollouts, instant rollbacks, and A/B testing without redeploying infrastructure.
Is NVIDIA Triton supported for high-throughput batch inference?
Yes. model-serving-kubernetes configures NVIDIA Triton for high-throughput batch inference with dynamic batching, model repositories on S3, and GPU optimization. Triton maximizes throughput by batching requests and supports multiple frameworks (PyTorch, TensorFlow). It's ideal when you need fine-grained control over inference performance.
What's the difference between KServe and Triton in model-serving-kubernetes?
model-serving-kubernetes supports both: KServe provides a higher-level abstraction with simpler setup, automatic model server selection, and built-in canary/traffic splitting. Triton offers lower-level control, dynamic batching, and multi-framework support for maximum throughput. Choose KServe for ease, Triton for performance tuning.
SKILL.md
Rendered from the published skill. Quoted content, verbatim.
Model Serving on Kubernetes
Production ML model serving with KServe and Triton — canary deployments, autoscaling, and GPU-aware scheduling.
When to Use This Skill
Use this skill when: - Serving scikit-learn, PyTorch, TensorFlow, or ONNX models at scale - Implementing canary deployments and A/B testing for ML models - Autoscaling inference pods based on request rate or GPU metrics - Deploying LLMs with Triton or KServe on Kubernetes - Managing multiple model versions with traffic splitting
Prerequisites
- Kubernetes 1.28+ with GPU nodes
- KServe installed (or Triton standalone)
kubectlandhelmconfigured- NVIDIA GPU Operator installed on cluster
KServe Installation
```bash
Install KServe with Helm
helm repo add kserve https://kserve.github.io/helm-charts helm repo update
helm install kserve kserve/kserve \
(truncated - see the full file via the links below)
File tree — 1 file
devops/orchestration/model-serving-kubernetes/SKILL.md
Let your AI agent find skills like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 56,283 agent skills by what they can do, searchable in plain language.
wish › “Deploy and serve ML models on Kubernetes at scale”
Give your agent the search over MCP, or paste the wish link into any chat. No install? Search from any chat →
Related skills
This skill enables dynamic scaling of LLM inference workloads across Kubernetes clusters using KEDA and Prometheus metrics tied to GPU utilization and request queues. It covers vLLM deployment, queue-based job scaling with Redis, spot instance strategies, and cluster autoscaler configuration to handle traffic spikes while optimizing costs.
This skill guides you through deploying machine learning workloads on managed cloud platforms, Kubernetes clusters, and serverless systems. It covers platform selection across AWS, GCP, Azure, Databricks, and specialized providers, plus practical patterns for endpoint configuration, training job orchestration, and scaling decisions based on your workload's latency, throughput, and compliance needs.
Build shared LLM inference platforms that safely serve multiple teams or customers without compromising security or performance. This skill covers tenant identity enforcement, per-tenant API keys, namespace isolation, quota configuration, request routing, and cost attribution—enabling you to run efficient, governed multi-tenant deployments on Kubernetes with vLLM and API gateways.
Deploy and operate production-grade GPU clusters in Kubernetes with built-in support for NVIDIA device plugins, MIG partitioning, and time-slicing. Monitor GPU health via DCGM metrics and Prometheus, configure autoscaling policies, and troubleshoot scheduling and driver issues across your AI infrastructure.
LLMOps Platform Engineering teaches you to architect internal LLM platforms that balance rapid experimentation with production safety. You'll implement model promotion pipelines with automated quality and safety gates, canary validation, and rollback capabilities, plus set up A/B testing infrastructure and observability across Kubernetes and cloud inference.
Azure AKS provisions managed Kubernetes clusters on Azure with built-in support for node pool configuration, network policies, and auto-scaling. The skill covers cluster creation, ingress setup, Container Insights monitoring, Azure Container Registry integration, and Terraform-based infrastructure as code for repeatable deployments.
More skills model-serving (MIT) · gpu-server-management (MIT) · aks-automatic-2025 (MIT) · arm-templates (MIT) · semantic-versioning (MIT)