model-serving-kubernetes
Run production ML inference on Kubernetes using KServe or NVIDIA Triton, with built-in support for canary traffic splitting, request-based autoscaling, and GPU resource allocation. The skill covers model versioning, A/B testing patterns, and dynamic batching for throughput optimization.
Model Serving on Kubernetes deploys ML models at scale using KServe and Triton with canary rollouts and autoscaling.
AI-generated summary based on this skill's SKILL.md
Install
BagelHole/DevOps-Security-Agent-Skills/model-serving-kubernetes · repository language: Shell
git clone https://github.com/BagelHole/DevOps-Security-Agent-Skills
cp -r DevOps-Security-Agent-Skills/devops/orchestration/model-serving-kubernetes ~/.claude/skills/model-serving-kubernetesnpx skillfed install BagelHole/DevOps-Security-Agent-Skills/model-serving-kubernetesFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I deploy machine learning models on Kubernetes?
model-serving-kubernetes enables production ML inference on Kubernetes using KServe or NVIDIA Triton. You define InferenceService resources that handle model deployment, versioning, and traffic routing. KServe abstracts Kubernetes complexity with built-in model server integration, while Triton offers lower-level control for high-throughput batch inference. Both support GPU allocation and automatic scaling based on request metrics.
Can model-serving-kubernetes implement canary deployments for ML models?
Yes. model-serving-kubernetes supports canary deployments through traffic splitting—route a percentage of requests to new model versions while keeping most traffic on stable versions. This enables safe rollouts and A/B testing of model updates. You configure traffic percentages in the InferenceService spec, allowing gradual validation before full promotion.
What autoscaling options does model-serving-kubernetes provide for GPU workloads?
model-serving-kubernetes autoscales inference workloads based on GPU and request metrics. It integrates with Kubernetes' Horizontal Pod Autoscaler to spawn inference pods when demand rises and shrink when idle. You set target metrics like request latency or GPU utilization, enabling cost-efficient scaling of expensive GPU resources.
How does model-serving-kubernetes handle multiple model versions in production?
model-serving-kubernetes manages multiple model versions through traffic splitting and canary patterns. Deploy different versions as separate predictors within a single InferenceService, then allocate traffic percentages to each. This supports gradual rollouts, instant rollbacks, and A/B testing without redeploying infrastructure.
Is NVIDIA Triton supported for high-throughput batch inference?
Yes. model-serving-kubernetes configures NVIDIA Triton for high-throughput batch inference with dynamic batching, model repositories on S3, and GPU optimization. Triton maximizes throughput by batching requests and supports multiple frameworks (PyTorch, TensorFlow). It's ideal when you need fine-grained control over inference performance.
What's the difference between KServe and Triton in model-serving-kubernetes?
model-serving-kubernetes supports both: KServe provides a higher-level abstraction with simpler setup, automatic model server selection, and built-in canary/traffic splitting. Triton offers lower-level control, dynamic batching, and multi-framework support for maximum throughput. Choose KServe for ease, Triton for performance tuning.
SKILL.md
rendered from the published skill — quoted content, verbatim
Model Serving on Kubernetes
Production ML model serving with KServe and Triton — canary deployments, autoscaling, and GPU-aware scheduling.
When to Use This Skill
Use this skill when: - Serving scikit-learn, PyTorch, TensorFlow, or ONNX models at scale - Implementing canary deployments and A/B testing for ML models - Autoscaling inference pods based on request rate or GPU metrics - Deploying LLMs with Triton or KServe on Kubernetes - Managing multiple model versions with traffic splitting
Prerequisites
- Kubernetes 1.28+ with GPU nodes
- KServe installed (or Triton standalone)
kubectlandhelmconfigured- NVIDIA GPU Operator installed on cluster
KServe Installation
```bash
Install KServe with Helm
helm repo add kserve https://kserve.github.io/helm-charts helm repo update
helm install kserve kserve/kserve \
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 1 file
devops/orchestration/model-serving-kubernetes/SKILL.md