vllm-server
vllm-server guides you through deploying and configuring vLLM—a high-performance open-source LLM serving engine—for production workloads. Set up continuous batching, multi-GPU tensor parallelism, model quantization, and OpenAI-compatible API endpoints to serve models like Llama and Mistral at scale. Includes Docker deployment, performance tuning, monitoring with Prometheus metrics, and troubleshooting for common VRAM and throughput issues.
vllm-server deploys production-grade LLM inference with continuous batching, tensor parallelism, and OpenAI-compatible APIs.
AI-generated summary based on this skill's SKILL.md
Decision gist · record as of 2026-05-22
vllm-server deploys production-grade LLM inference with continuous batching, tensor parallelism, and OpenAI-compatible APIs. vllm-server guides you through deploying and configuring vLLM—a high-performance open-source LLM serving engine—for production workloads. Set up continuous batching, multi-GPU tensor parallelism, model quantization, and OpenAI-compatible API endpoints to serve models like Llama and Mistral at scale. Includes Docker deployment, performance tuning, monitoring with Prometheus metrics, and troubleshooting for common VRAM and throughput issues.
Use it when
- vllm-server covers tensor parallelism configuration to distribute model inference across multiple GPUs.
- vllm-server documents quantization techniques including AWQ and GPTQ to reduce GPU memory requirements.
Verify before relying
Read SKILL.md below before installing (1 file). Open directory: indexed for reading, not audited.
Install
BagelHole/DevOps-Security-Agent-Skills/vllm-server · repository language: Shell
Open directory. Skills are indexed for reading, not audited. Review a skill's body before installing it.
Frequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I deploy vllm server for production LLM inference?
vllm-server helps you deploy vLLM—a high-performance open-source LLM serving engine—for production workloads. Start by installing vLLM, then configure your model, GPU allocation, and batch settings. Use Docker for containerized deployment or Kubernetes for orchestrated multi-node setups. vllm-server guides you through setting up continuous batching and paged attention to maximize throughput, configuring tensor parallelism across multiple GPUs, and exposing OpenAI-compatible API endpoints for seamless integration with existing applications.
How do I run vllm with multiple GPUs using tensor parallelism?
vllm-server covers tensor parallelism configuration to distribute model inference across multiple GPUs. Enable tensor parallelism by specifying the number of GPUs and partition strategy in your vLLM launch command. vllm-server provides step-by-step guidance on configuring GPU allocation, setting tensor-parallel size, and validating that your model shards correctly across devices. This approach significantly reduces per-GPU memory requirements and enables serving larger models while maintaining low latency.
What quantization methods does vllm-server support for memory optimization?
vllm-server documents quantization techniques including AWQ and GPTQ to reduce GPU memory requirements. These methods compress model weights while maintaining inference quality, allowing you to serve larger models or fit more concurrent requests on available VRAM. vllm-server explains how to load pre-quantized models, configure quantization parameters, and measure the trade-offs between memory savings and accuracy for your workload.
How do I set up an OpenAI-compatible API endpoint with vllm-server?
vllm-server guides you through configuring OpenAI-compatible API endpoints so your self-hosted models work with existing client libraries and applications. Launch vLLM with the API server enabled, specify your model and port, and vllm-server shows you how to test endpoints using standard OpenAI client code. This compatibility layer lets you swap between cloud providers and self-hosted inference without changing application code.
How do I monitor vllm performance and troubleshoot resource issues?
vllm-server provides monitoring guidance including Prometheus metrics integration to track throughput, latency, GPU utilization, and memory consumption. It covers troubleshooting common issues like CUDA out-of-memory errors, identifying bottlenecks, and tuning batch size and context window settings. vllm-server helps you correlate metrics with performance problems and adjust configurations—such as reducing max model length or enabling quantization—to resolve resource constraints.
Can vllm-server help optimize throughput and latency for high-volume inference?
vllm-server focuses on throughput and latency optimization through continuous batching, paged attention, tensor parallelism, and quantization. It explains how to tune batch size, prefill/decode ratios, and GPU memory allocation to balance request concurrency with response time. vllm-server includes benchmarking guidance to measure improvements and configuration recommendations for models like Llama and Mistral serving high-volume production traffic.
SKILL.md
Rendered from the published skill. Quoted content, verbatim.
vLLM Server Management
Deploy production-grade LLM inference servers with vLLM — the fastest open-source LLM serving engine with PagedAttention and continuous batching.
When to Use This Skill
Use this skill when: - Serving open-source LLMs (Llama, Mistral, Qwen, Gemma) at scale - Building an OpenAI-compatible API endpoint for self-hosted models - Optimizing LLM throughput and latency for production traffic - Running multi-GPU inference with tensor or pipeline parallelism - Deploying quantized models to reduce GPU memory requirements
Prerequisites
- NVIDIA GPU(s) with CUDA 12.1+ (A100/H100 recommended for production)
- Docker or Python 3.9+ with pip
- 40GB+ VRAM for 70B models; 8GB+ for 7B models
nvidia-container-toolkitfor Docker GPU passthrough
Quick Start
```bash
Install vLLM
pip install vllm
Serve a model (OpenAI-compatible API)
vllm serve
(truncated - see the full file via the links below)
File tree — 1 file
infrastructure/local-ai/vllm-server/SKILL.md
Let your AI agent find skills like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 56,283 agent skills by what they can do, searchable in plain language.
wish › “Deploy and configure vLLM for production LLM inference at scale”
Give your agent the search over MCP, or paste the wish link into any chat. No install? Search from any chat →
Related skills
serving-llms-vllm accelerates LLM inference for production environments through PagedAttention-based memory optimization and continuous batching. It supports OpenAI-compatible endpoints, quantization methods like GPTQ and AWQ, and tensor parallelism across multiple GPUs. Use this skill when deploying scalable LLM services that demand both low latency and high request throughput.
vLLM runs open-weight models on local GPUs with high-throughput inference behind an OpenAI-compatible endpoint, ready for chat and agent workloads. Configure context length and quantization to fit your hardware, then register the endpoint with PenguinHarness to make it available to your agents.
Model Serving equips you to run LLMs and traditional ML models in production using optimized inference engines like vLLM and BentoML. It covers GPU memory optimization, continuous batching, streaming response patterns, and integration with frontend applications for real-world AI deployments.
vLLM accelerates LLM inference for production deployments using PagedAttention and continuous batching to achieve significantly higher throughput than standard approaches. It provides OpenAI-compatible endpoints, supports quantization methods like AWQ and GPTQ for memory-constrained setups, and enables tensor parallelism across multiple GPUs. Deploy via Docker, monitor with Prometheus metrics, and handle batch processing or real-time serving workflows.
A structured decision guide for deploying and tuning vLLM in production environments. Covers the PagedAttention memory model, continuous batching, quantization tradeoffs, tensor/pipeline parallelism choices, and step-by-step operational workflows for diagnosing throughput, latency, and out-of-memory issues. Includes comparisons to alternative inference engines and guidance on when vLLM is the right fit.
GPU Server Management guides you through provisioning and configuring NVIDIA GPU infrastructure for LLM inference and model training. It covers driver and CUDA toolkit installation, Docker GPU integration, multi-GPU topology setup, and production monitoring with DCGM and Prometheus metrics.
More skills llmops-platform-engineering (MIT) · semantic-versioning (MIT) · arm-templates (MIT) · ollama (Apache-2.0) · llm-fine-tuning (MIT)