agentsop-vllm
A structured decision guide for deploying and tuning vLLM in production environments. Covers the PagedAttention memory model, continuous batching, quantization tradeoffs, tensor/pipeline parallelism choices, and step-by-step operational workflows for diagnosing throughput, latency, and out-of-memory issues. Includes comparisons to alternative inference engines and guidance on when vLLM is the right fit.
agentsop-vllm guides GPU-backed LLM serving decisions, from model selection through parallelism tuning and OOM triage.
AI-generated summary based on this skill's SKILL.md
Install
agentsope/SkillAlchemy/agentsop-vllm · repository language: Python
git clone https://github.com/agentsope/SkillAlchemy
cp -r SkillAlchemy/skills/agentsop-vllm ~/.claude/skills/agentsop-vllmnpx skillfed install agentsope/SkillAlchemy/agentsop-vllmFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I serve Llama 70B with vLLM on production GPUs?
agentsop-vLLM handles large-model serving through tensor parallelism and quantization. For Llama 70B, start by enabling tensor parallelism across multiple GPUs (e.g., `--tensor-parallel-size 4` on four H100s). Apply quantization—AWQ or FP8—to reduce memory footprint by 4–8×. Set `--max-model-len` conservatively to avoid OOM during inference. Use continuous batching to maximize throughput. Monitor KV cache fragmentation; if severe, enable prefix caching for repeated prompts or reduce batch size to trade throughput for latency stability.
What's the difference between vLLM's FP8 and AWQ quantization?
agentsop-vLLM supports both post-training quantization methods with different tradeoffs. FP8 quantizes weights and activations symmetrically, offering faster inference but slightly lower accuracy; it works well when latency is critical. AWQ (Activation-aware Weight Quantization) preserves accuracy better by protecting salient weights, making it preferable when quality matters more than speed. Choose FP8 for high-throughput batch serving; choose AWQ when you need tighter accuracy margins. Test both on your workload—accuracy loss varies by model and task.
How do I fix vLLM CUDA out-of-memory errors?
agentsop-vLLM OOM errors typically stem from KV cache growth or oversized batch sizes. First, reduce `--max-model-len` to limit context length per request. Second, lower `--max-num-batched-tokens` to shrink the batch window. Third, enable quantization (FP8 or AWQ) to halve memory use. Fourth, apply tensor parallelism to distribute weights across GPUs. If issues persist, check for memory leaks via `nvidia-smi` over time. Enable prefix caching to reuse KV for repeated prefixes. As a last resort, reduce batch size or switch to pipeline parallelism to trade latency for stability.
How should I set up tensor parallelism and pipeline parallelism in vLLM?
agentsop-vLLM's parallelism strategy depends on model size and GPU count. Tensor parallelism splits each layer across GPUs—ideal for models larger than single-GPU memory. Use `--tensor-parallel-size N` where N divides the model's layer count evenly. Pipeline parallelism splits layers sequentially across GPUs—better for very deep models or when tensor parallelism causes communication bottlenecks. Combine both for extreme scale. Tensor parallelism is simpler and recommended first; pipeline parallelism adds latency due to bubble overhead. Benchmark both on your hardware to find the sweet spot.
When should I use vLLM prefix caching versus continuous batching alone?
agentsop-vLLM's prefix caching reuses KV cache blocks for identical prompt prefixes, cutting memory and compute. Enable it when serving repeated system prompts, few-shot examples, or document retrieval tasks. Continuous batching alone works well for diverse, one-off requests. Prefix caching shines in chatbots, RAG systems, and multi-turn conversations where prefixes overlap. The overhead is minimal—enable it by default unless your workload has zero prefix reuse. Combine it with chunked prefill to prevent head-of-line blocking when processing long shared prefixes.
How does vLLM compare to TensorRT-LLM and SGLang for production inference?
agentsop-vLLM excels at ease of deployment and broad model support; it's production-ready out of the box with minimal tuning. TensorRT-LLM offers lower latency and higher throughput for specific models (Llama, Mistral) via aggressive kernel optimization, but requires more setup and model-specific plugins. SGLang adds structured generation and better latency for complex prompts but is newer and less battle-tested. Choose vLLM for fast time-to-market and model diversity; choose TensorRT-LLM if you need maximum performance on supported models; choose SGLang if you need structured output or complex control flow. vLLM's PagedAttention and continuous batching make it the safest default for most teams.
SKILL.md
rendered from the published skill — quoted content, verbatim
vLLM Serving SOP
1. 何时激活 (When to activate)
Activate this skill when any of the following hold:
- The user wants to serve an LLM in production (multi-user, concurrent requests, throughput-oriented) and has GPU infrastructure.
- The user is comparing inference engines (vLLM vs TGI vs SGLang vs TensorRT-LLM vs llama.cpp/Ollama).
- The user reports a vLLM operational issue: CUDA OOM, low throughput, high TTFT, request preemption, multi-GPU setup, quantization choice.
- The user is sizing hardware for an open-weights model (Llama /
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 8 files
skills/agentsop-vllm/README.md
skills/agentsop-vllm/SKILL.md
skills/agentsop-vllm/intermediate/operation_candidates.json
skills/agentsop-vllm/references/R1-architecture.md
skills/agentsop-vllm/references/R2-sop-workflow.md
skills/agentsop-vllm/references/R3-dilemma-cases.md
skills/agentsop-vllm/references/R4-anti-patterns.md
skills/agentsop-vllm/references/R5-ecosystem-context.md