gpu-kubernetes-operations
Deploy and operate production-grade GPU clusters in Kubernetes with built-in support for NVIDIA device plugins, MIG partitioning, and time-slicing. Monitor GPU health via DCGM metrics and Prometheus, configure autoscaling policies, and troubleshoot scheduling and driver issues across your AI infrastructure.
GPU Kubernetes Operations helps you set up and manage GPU-backed Kubernetes clusters for AI inference and training workloads.
AI-generated summary based on this skill's SKILL.md
Install
BagelHole/DevOps-Security-Agent-Skills/gpu-kubernetes-operations · repository language: Shell
git clone https://github.com/BagelHole/DevOps-Security-Agent-Skills
cp -r DevOps-Security-Agent-Skills/infrastructure/local-ai/gpu-kubernetes-operations ~/.claude/skills/gpu-kubernetes-operationsnpx skillfed install BagelHole/DevOps-Security-Agent-Skills/gpu-kubernetes-operationsFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How to set up GPU nodes in Kubernetes?
gpu-kubernetes-operations enables GPU node setup through NVIDIA device plugins and the GPU Operator. Install the NVIDIA GPU Operator on your cluster to automatically deploy drivers, container toolkit, and device plugins across GPU nodes. Configure node pools with GPU labels and taints, then schedule AI workloads using GPU resource requests (nvidia.com/gpu). The skill covers full lifecycle management from driver installation through pod scheduling.
What is MIG partitioning and how do I use it in Kubernetes?
gpu-kubernetes-operations supports Multi-Instance GPU (MIG) partitioning for A100 and H100 GPUs, allowing you to divide a single GPU into multiple isolated instances. Enable MIG mode on your GPU nodes, partition GPUs into compute instances, and configure the device plugin to expose partitions as separate resources. This maximizes utilization by running multiple smaller workloads simultaneously on one physical GPU.
How can I monitor GPU health and metrics with DCGM in Kubernetes?
gpu-kubernetes-operations integrates DCGM (Data Center GPU Manager) for comprehensive GPU health monitoring. Deploy DCGM exporter as a DaemonSet to collect metrics like temperature, power, memory usage, and error counts. Scrape metrics into Prometheus and set up alerts for GPU failures, thermal issues, and XID errors. This enables proactive detection of hardware problems before they impact your AI workloads.
How do I troubleshoot GPU pod pending issues in Kubernetes?
gpu-kubernetes-operations provides troubleshooting guidance for GPU scheduling failures. Check device plugin logs, verify GPU resource availability with `kubectl describe nodes`, and confirm pod requests match available GPU types. Common causes include driver mismatches, insufficient GPU memory, node taints without matching tolerations, or device plugin crashes. Use DCGM health checks to rule out hardware failures.
What GPU time-slicing setup does this skill support?
gpu-kubernetes-operations enables GPU time-slicing to share a single GPU across multiple pods sequentially. Configure the device plugin with sharing mode, set time-slice durations, and define fairness policies. This allows cost-effective multi-tenant AI inference clusters where pods take turns accessing GPU compute. Combine with MIG partitioning for fine-grained resource isolation and higher throughput.
How can I implement GPU autoscaling and cost optimization?
gpu-kubernetes-operations supports GPU autoscaling through Kubernetes HPA and cluster autoscaler integration. Define GPU resource requests in pod specs, set HPA scaling policies based on GPU utilization metrics, and configure node auto-provisioning to add GPU nodes on demand. Combine with time-slicing and MIG to maximize utilization, reducing idle GPU costs while maintaining performance for your AI inference and training workloads.
SKILL.md
rendered from the published skill — quoted content, verbatim
GPU Kubernetes Operations
Run resilient and cost-efficient GPU clusters for production AI workloads.
When to Use This Skill
- Setting up GPU node pools in Kubernetes for AI inference or training
- Configuring NVIDIA device plugin and GPU operator
- Implementing MIG partitioning to share GPUs across workloads
- Building GPU-aware autoscaling policies
- Monitoring GPU health with DCGM and Prometheus
- Troubleshooting GPU scheduling, driver, or OOM issues
Prerequisites
- Kubernetes 1.28+ cluster with GPU-capable nodes
- NVIDIA GPUs (A10, L4, A100, H100, or similar)
- NVIDIA drivers installed on nodes (535+ recommended)
- Helm 3 for operator and plugin installation
- Prometheus stack for metrics collection
NVIDIA GPU Operator Installation
The GPU Operator automates driver, toolkit, device plugin, and DCGM deployment.
```bash
Add NVIDIA Helm repo
helm repo add nvidia
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 1 file
infrastructure/local-ai/gpu-kubernetes-operations/SKILL.md