{"enrichment":{"faq":[{"a":"agentsop-vLLM handles large-model serving through tensor parallelism and quantization. For Llama 70B, start by enabling tensor parallelism across multiple GPUs (e.g., `--tensor-parallel-size 4` on four H100s). Apply quantization\u2014AWQ or FP8\u2014to reduce memory footprint by 4\u20138\u00d7. Set `--max-model-len` conservatively to avoid OOM during inference. Use continuous batching to maximize throughput. Monitor KV cache fragmentation; if severe, enable prefix caching for repeated prompts or reduce batch size to trade throughput for latency stability.","q":"How do I serve Llama 70B with vLLM on production GPUs?"},{"a":"agentsop-vLLM supports both post-training quantization methods with different tradeoffs. FP8 quantizes weights and activations symmetrically, offering faster inference but slightly lower accuracy; it works well when latency is critical. AWQ (Activation-aware Weight Quantization) preserves accuracy better by protecting salient weights, making it preferable when quality matters more than speed. Choose FP8 for high-throughput batch serving; choose AWQ when you need tighter accuracy margins. Test both on your workload\u2014accuracy loss varies by model and task.","q":"What's the difference between vLLM's FP8 and AWQ quantization?"},{"a":"agentsop-vLLM OOM errors typically stem from KV cache growth or oversized batch sizes. First, reduce `--max-model-len` to limit context length per request. Second, lower `--max-num-batched-tokens` to shrink the batch window. Third, enable quantization (FP8 or AWQ) to halve memory use. Fourth, apply tensor parallelism to distribute weights across GPUs. If issues persist, check for memory leaks via `nvidia-smi` over time. Enable prefix caching to reuse KV for repeated prefixes. As a last resort, reduce batch size or switch to pipeline parallelism to trade latency for stability.","q":"How do I fix vLLM CUDA out-of-memory errors?"},{"a":"agentsop-vLLM's parallelism strategy depends on model size and GPU count. Tensor parallelism splits each layer across GPUs\u2014ideal for models larger than single-GPU memory. Use `--tensor-parallel-size N` where N divides the model's layer count evenly. Pipeline parallelism splits layers sequentially across GPUs\u2014better for very deep models or when tensor parallelism causes communication bottlenecks. Combine both for extreme scale. Tensor parallelism is simpler and recommended first; pipeline parallelism adds latency due to bubble overhead. Benchmark both on your hardware to find the sweet spot.","q":"How should I set up tensor parallelism and pipeline parallelism in vLLM?"},{"a":"agentsop-vLLM's prefix caching reuses KV cache blocks for identical prompt prefixes, cutting memory and compute. Enable it when serving repeated system prompts, few-shot examples, or document retrieval tasks. Continuous batching alone works well for diverse, one-off requests. Prefix caching shines in chatbots, RAG systems, and multi-turn conversations where prefixes overlap. The overhead is minimal\u2014enable it by default unless your workload has zero prefix reuse. Combine it with chunked prefill to prevent head-of-line blocking when processing long shared prefixes.","q":"When should I use vLLM prefix caching versus continuous batching alone?"},{"a":"agentsop-vLLM excels at ease of deployment and broad model support; it's production-ready out of the box with minimal tuning. TensorRT-LLM offers lower latency and higher throughput for specific models (Llama, Mistral) via aggressive kernel optimization, but requires more setup and model-specific plugins. SGLang adds structured generation and better latency for complex prompts but is newer and less battle-tested. Choose vLLM for fast time-to-market and model diversity; choose TensorRT-LLM if you need maximum performance on supported models; choose SGLang if you need structured output or complex control flow. vLLM's PagedAttention and continuous batching make it the safest default for most teams.","q":"How does vLLM compare to TensorRT-LLM and SGLang for production inference?"}],"shadow_tags":["gpu-inference-serving","kv-cache-management","model-quantization","distributed-parallelism","batch-scheduling","production-deployment","latency-optimization","throughput-scaling","memory-efficiency","inference-engines"],"summary_rewrite":"A structured decision guide for deploying and tuning vLLM in production environments. Covers the PagedAttention memory model, continuous batching, quantization tradeoffs, tensor/pipeline parallelism choices, and step-by-step operational workflows for diagnosing throughput, latency, and out-of-memory issues. Includes comparisons to alternative inference engines and guidance on when vLLM is the right fit."},"files":[{"bytes":24676,"path":"skills/agentsop-vllm/SKILL.md","sha256":"11749c55b3ff0c364d40fd8bea96583a9fa8d1ac193bcde02dc668cebdaee0d7","url":"https://skillfed.io/files/agentsope/SkillAlchemy/agentsop-vllm/81b8cacc/SKILL.md"}],"id":"agentsope/SkillAlchemy/agentsop-vllm","links":{"html":"https://skillfed.io/agentsope/SkillAlchemy/agentsop-vllm","md":"https://skillfed.io/agentsope/SkillAlchemy/agentsop-vllm.md","repo":"https://github.com/agentsope/SkillAlchemy"},"meta":{"agents_supported":[],"first_seen":"2026-07-28","forks":12,"language":"Python","last_updated":"2026-06-30","license":"MIT","name":"agentsop-vllm","publisher":"agentsope","stars":219},"relations":{"similar":[{"id":"agentsope/SkillAlchemy/agentsop-llm-engine-selection"},{"id":"vasilyu1983/AI-Agents-public/ai-llm-inference"},{"id":"moltis-org/moltis/serving-llms-vllm"},{"id":"Orchestra-Research/AI-Research-SKILLs/vllm"},{"id":"OpenLAIR/dr-claw/vllm"},{"id":"synthetic-sciences/openscience/vllm"},{"id":"NousResearch/hermes-agent/serving-llms-vllm"},{"id":"graniet/kheish/vllm"},{"id":"synthetic-sciences/openscience/tensorrt-llm"},{"id":"Orchestra-Research/AI-Research-SKILLs/tensorrt-llm"}]},"slug":{"owner":"agentsope","repo":"SkillAlchemy","skill":"agentsop-vllm"},"version":"81b8cacc"}