vllm-server
vllm-server guides you through deploying and configuring vLLM—a high-performance open-source LLM serving engine—for production workloads. Set up continuous batching, multi-GPU tensor parallelism, model quantization, and OpenAI-compatible API endpoints to serve models like Llama and Mistral at scale. Includes Docker deployment, performance tuning, monitoring with Prometheus metrics, and troubleshooting for common VRAM and throughput issues.
vllm-server deploys production-grade LLM inference with continuous batching, tensor parallelism, and OpenAI-compatible APIs.
AI-generated summary based on this skill's SKILL.md
Install
BagelHole/DevOps-Security-Agent-Skills/vllm-server · repository language: Shell
git clone https://github.com/BagelHole/DevOps-Security-Agent-Skills
cp -r DevOps-Security-Agent-Skills/infrastructure/local-ai/vllm-server ~/.claude/skills/vllm-servernpx skillfed install BagelHole/DevOps-Security-Agent-Skills/vllm-serverFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I deploy vllm server for production LLM inference?
vllm-server helps you deploy vLLM—a high-performance open-source LLM serving engine—for production workloads. Start by installing vLLM, then configure your model, GPU allocation, and batch settings. Use Docker for containerized deployment or Kubernetes for orchestrated multi-node setups. vllm-server guides you through setting up continuous batching and paged attention to maximize throughput, configuring tensor parallelism across multiple GPUs, and exposing OpenAI-compatible API endpoints for seamless integration with existing applications.
How do I run vllm with multiple GPUs using tensor parallelism?
vllm-server covers tensor parallelism configuration to distribute model inference across multiple GPUs. Enable tensor parallelism by specifying the number of GPUs and partition strategy in your vLLM launch command. vllm-server provides step-by-step guidance on configuring GPU allocation, setting tensor-parallel size, and validating that your model shards correctly across devices. This approach significantly reduces per-GPU memory requirements and enables serving larger models while maintaining low latency.
What quantization methods does vllm-server support for memory optimization?
vllm-server documents quantization techniques including AWQ and GPTQ to reduce GPU memory requirements. These methods compress model weights while maintaining inference quality, allowing you to serve larger models or fit more concurrent requests on available VRAM. vllm-server explains how to load pre-quantized models, configure quantization parameters, and measure the trade-offs between memory savings and accuracy for your workload.
How do I set up an OpenAI-compatible API endpoint with vllm-server?
vllm-server guides you through configuring OpenAI-compatible API endpoints so your self-hosted models work with existing client libraries and applications. Launch vLLM with the API server enabled, specify your model and port, and vllm-server shows you how to test endpoints using standard OpenAI client code. This compatibility layer lets you swap between cloud providers and self-hosted inference without changing application code.
How do I monitor vllm performance and troubleshoot resource issues?
vllm-server provides monitoring guidance including Prometheus metrics integration to track throughput, latency, GPU utilization, and memory consumption. It covers troubleshooting common issues like CUDA out-of-memory errors, identifying bottlenecks, and tuning batch size and context window settings. vllm-server helps you correlate metrics with performance problems and adjust configurations—such as reducing max model length or enabling quantization—to resolve resource constraints.
Can vllm-server help optimize throughput and latency for high-volume inference?
vllm-server focuses on throughput and latency optimization through continuous batching, paged attention, tensor parallelism, and quantization. It explains how to tune batch size, prefill/decode ratios, and GPU memory allocation to balance request concurrency with response time. vllm-server includes benchmarking guidance to measure improvements and configuration recommendations for models like Llama and Mistral serving high-volume production traffic.
SKILL.md
rendered from the published skill — quoted content, verbatim
vLLM Server Management
Deploy production-grade LLM inference servers with vLLM — the fastest open-source LLM serving engine with PagedAttention and continuous batching.
When to Use This Skill
Use this skill when: - Serving open-source LLMs (Llama, Mistral, Qwen, Gemma) at scale - Building an OpenAI-compatible API endpoint for self-hosted models - Optimizing LLM throughput and latency for production traffic - Running multi-GPU inference with tensor or pipeline parallelism - Deploying quantized models to reduce GPU memory requirements
Prerequisites
- NVIDIA GPU(s) with CUDA 12.1+ (A100/H100 recommended for production)
- Docker or Python 3.9+ with pip
- 40GB+ VRAM for 70B models; 8GB+ for 7B models
nvidia-container-toolkitfor Docker GPU passthrough
Quick Start
```bash
Install vLLM
pip install vllm
Serve a model (OpenAI-compatible API)
vllm serve
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 1 file
infrastructure/local-ai/vllm-server/SKILL.md