agentsop-vllm
A structured decision guide for deploying and tuning vLLM in production environments. Covers the PagedAttention memory model, continuous batching, quantization tradeoffs, tensor/pipeline parallelism choices, and step-by-step operational workflows for diagnosing throughput, latency, and out-of-memory issues. Includes comparisons to alternative inference engines and guidance on when vLLM is the right fit.
agentsop-vllm guides GPU-backed LLM serving decisions, from model selection through parallelism tuning and OOM triage.
AI-generated summary based on this skill's SKILL.md
Decision gist · record as of 2026-06-30
agentsop-vllm guides GPU-backed LLM serving decisions, from model selection through parallelism tuning and OOM triage. A structured decision guide for deploying and tuning vLLM in production environments. Covers the PagedAttention memory model, continuous batching, quantization tradeoffs, tensor/pipeline parallelism choices, and step-by-step operational workflows for diagnosing throughput, latency, and out-of-memory issues. Includes comparisons to alternative inference engines and guidance on when vLLM is the right fit.
Use it when
- agentsop-vLLM supports both post-training quantization methods with different tradeoffs.
- agentsop-vLLM OOM errors typically stem from KV cache growth or oversized batch sizes.
Verify before relying
Read SKILL.md below before installing (8 files). Open directory: indexed for reading, not audited.
Install
agentsope/SkillAlchemy/agentsop-vllm · repository language: Python
Open directory. Skills are indexed for reading, not audited. Review a skill's body before installing it.
Frequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I serve Llama 70B with vLLM on production GPUs?
agentsop-vLLM handles large-model serving through tensor parallelism and quantization. For Llama 70B, start by enabling tensor parallelism across multiple GPUs (e.g., `--tensor-parallel-size 4` on four H100s). Apply quantization—AWQ or FP8—to reduce memory footprint by 4–8×. Set `--max-model-len` conservatively to avoid OOM during inference. Use continuous batching to maximize throughput. Monitor KV cache fragmentation; if severe, enable prefix caching for repeated prompts or reduce batch size to trade throughput for latency stability.
What's the difference between vLLM's FP8 and AWQ quantization?
agentsop-vLLM supports both post-training quantization methods with different tradeoffs. FP8 quantizes weights and activations symmetrically, offering faster inference but slightly lower accuracy; it works well when latency is critical. AWQ (Activation-aware Weight Quantization) preserves accuracy better by protecting salient weights, making it preferable when quality matters more than speed. Choose FP8 for high-throughput batch serving; choose AWQ when you need tighter accuracy margins. Test both on your workload—accuracy loss varies by model and task.
How do I fix vLLM CUDA out-of-memory errors?
agentsop-vLLM OOM errors typically stem from KV cache growth or oversized batch sizes. First, reduce `--max-model-len` to limit context length per request. Second, lower `--max-num-batched-tokens` to shrink the batch window. Third, enable quantization (FP8 or AWQ) to halve memory use. Fourth, apply tensor parallelism to distribute weights across GPUs. If issues persist, check for memory leaks via `nvidia-smi` over time. Enable prefix caching to reuse KV for repeated prefixes. As a last resort, reduce batch size or switch to pipeline parallelism to trade latency for stability.
How should I set up tensor parallelism and pipeline parallelism in vLLM?
agentsop-vLLM's parallelism strategy depends on model size and GPU count. Tensor parallelism splits each layer across GPUs—ideal for models larger than single-GPU memory. Use `--tensor-parallel-size N` where N divides the model's layer count evenly. Pipeline parallelism splits layers sequentially across GPUs—better for very deep models or when tensor parallelism causes communication bottlenecks. Combine both for extreme scale. Tensor parallelism is simpler and recommended first; pipeline parallelism adds latency due to bubble overhead. Benchmark both on your hardware to find the sweet spot.
When should I use vLLM prefix caching versus continuous batching alone?
agentsop-vLLM's prefix caching reuses KV cache blocks for identical prompt prefixes, cutting memory and compute. Enable it when serving repeated system prompts, few-shot examples, or document retrieval tasks. Continuous batching alone works well for diverse, one-off requests. Prefix caching shines in chatbots, RAG systems, and multi-turn conversations where prefixes overlap. The overhead is minimal—enable it by default unless your workload has zero prefix reuse. Combine it with chunked prefill to prevent head-of-line blocking when processing long shared prefixes.
How does vLLM compare to TensorRT-LLM and SGLang for production inference?
agentsop-vLLM excels at ease of deployment and broad model support; it's production-ready out of the box with minimal tuning. TensorRT-LLM offers lower latency and higher throughput for specific models (Llama, Mistral) via aggressive kernel optimization, but requires more setup and model-specific plugins. SGLang adds structured generation and better latency for complex prompts but is newer and less battle-tested. Choose vLLM for fast time-to-market and model diversity; choose TensorRT-LLM if you need maximum performance on supported models; choose SGLang if you need structured output or complex control flow. vLLM's PagedAttention and continuous batching make it the safest default for most teams.
SKILL.md
Rendered from the published skill. Quoted content, verbatim.
vLLM Serving SOP
1. 何时激活 (When to activate)
Activate this skill when any of the following hold:
- The user wants to serve an LLM in production (multi-user, concurrent requests, throughput-oriented) and has GPU infrastructure.
- The user is comparing inference engines (vLLM vs TGI vs SGLang vs TensorRT-LLM vs llama.cpp/Ollama).
- The user reports a vLLM operational issue: CUDA OOM, low throughput, high TTFT, request preemption, multi-GPU setup, quantization choice.
- The user is sizing hardware for an open-weights model (Llama /
(truncated - see the full file via the links below)
File tree — 8 files
skills/agentsop-vllm/README.md
skills/agentsop-vllm/SKILL.md
skills/agentsop-vllm/intermediate/operation_candidates.json
skills/agentsop-vllm/references/R1-architecture.md
skills/agentsop-vllm/references/R2-sop-workflow.md
skills/agentsop-vllm/references/R3-dilemma-cases.md
skills/agentsop-vllm/references/R4-anti-patterns.md
skills/agentsop-vllm/references/R5-ecosystem-context.md
Let your AI agent find skills like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 56,283 agent skills by what they can do, searchable in plain language.
wish › “Deploy and optimize LLM inference on GPUs using vLLM for production workloads”
Give your agent the search over MCP, or paste the wish link into any chat. No install? Search from any chat →
Related skills
This skill guides you through selecting an LLM serving engine by analyzing hardware topology, workload profile, and operational constraints rather than benchmark headlines. It maps seven engines—vLLM, SGLang, TensorRT-LLM, TGI, llama.cpp, Ollama, and MLX—to their strengths across GPU clusters, edge devices, and single-user scenarios, then walks you through a five-step decision workflow to eliminate incompatible options and benchmark your top candidates.
This skill delivers operational patterns for tuning LLM inference performance, cost, and reliability in production. It covers latency budgeting, continuous batching, KV-cache optimization, speculative decoding, quantization strategies, and parallelism choices—with decision trees and checklists to guide infrastructure and serving stack tuning.
vllm-server guides you through deploying and configuring vLLM—a high-performance open-source LLM serving engine—for production workloads. Set up continuous batching, multi-GPU tensor parallelism, model quantization, and OpenAI-compatible API endpoints to serve models like Llama and Mistral at scale. Includes Docker deployment, performance tuning, monitoring with Prometheus metrics, and troubleshooting for common VRAM and throughput issues.
serving-llms-vllm accelerates LLM inference for production environments through PagedAttention-based memory optimization and continuous batching. It supports OpenAI-compatible endpoints, quantization methods like GPTQ and AWQ, and tensor parallelism across multiple GPUs. Use this skill when deploying scalable LLM services that demand both low latency and high request throughput.
vLLM accelerates LLM inference for production deployments using PagedAttention and continuous batching to achieve significantly higher throughput than standard approaches. It provides OpenAI-compatible endpoints, supports quantization methods like AWQ and GPTQ for memory-constrained setups, and enables tensor parallelism across multiple GPUs. Deploy via Docker, monitor with Prometheus metrics, and handle batch processing or real-time serving workflows.
TensorRT-LLM accelerates large language model inference on NVIDIA GPUs through advanced optimization techniques including quantization, in-flight batching, and multi-GPU parallelism. Achieve production-grade throughput and latency for real-time applications with support for 100+ models.
More skills vllm (Apache-2.0) · gpu-server-management (MIT) · model-serving (MIT)