serving-llms-vllm
vLLM accelerates LLM inference for production deployments using PagedAttention and continuous batching to achieve significantly higher throughput than standard approaches. It provides OpenAI-compatible endpoints, supports quantization methods like AWQ and GPTQ for memory-constrained setups, and enables tensor parallelism across multiple GPUs. Deploy via Docker, monitor with Prometheus metrics, and handle batch processing or real-time serving workflows.
serving-llms-vllm enables production LLM API deployment with 24x higher throughput than standard transformers through PagedAttention and continuous batching.
AI-generated summary based on this skill's SKILL.md
Install
NousResearch/hermes-agent/serving-llms-vllm · repository language: Python
git clone https://github.com/NousResearch/hermes-agent
cp -r hermes-agent/skills/mlops/inference/serving-llms-vllm ~/.claude/skills/serving-llms-vllmnpx skillfed install NousResearch/hermes-agent/serving-llms-vllmFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How does vLLM achieve high throughput LLM serving?
vLLM accelerates LLM inference through PagedAttention, which reduces memory fragmentation by storing attention key-value caches in non-contiguous memory blocks. Combined with continuous batching, vLLM dynamically schedules requests to maximize GPU utilization. This architecture enables significantly higher throughput than standard approaches while maintaining low latency, making it ideal for production deployments handling variable request volumes.
What quantization methods does vLLM support to reduce memory usage?
vLLM supports quantization methods including AWQ and GPTQ to compress models and reduce memory footprint. These techniques allow you to serve large language models on limited GPU memory by reducing model size while maintaining reasonable inference quality. Quantization is particularly valuable when deploying large models like 70B-parameter variants on constrained hardware.
Can vLLM be deployed with OpenAI API compatibility?
Yes, vLLM provides OpenAI-compatible endpoints out of the box. You can set up vLLM servers that expose the same API interface as OpenAI's services, allowing existing applications and clients to switch to vLLM without code changes. This compatibility simplifies integration into production systems and enables drop-in replacement scenarios.
How does vLLM tensor parallelism work across multiple GPUs?
vLLM tensor parallelism distributes model computations across multiple GPUs, splitting tensors along specific dimensions so each GPU handles a portion of the computation. This approach enables serving very large models that exceed single-GPU memory capacity. Combined with other optimization techniques, tensor parallelism helps achieve both high throughput and low latency in multi-GPU setups.
What deployment options are available for vLLM in production?
vLLM supports Docker containerization for consistent deployments and integrates with Kubernetes for orchestration at scale. You can monitor performance using Prometheus metrics to track throughput, latency, and resource utilization. These deployment patterns enable reliable, observable production LLM serving with support for load testing and performance benchmarking.
How does vLLM handle batch inference on large datasets?
vLLM's continuous batching scheduler efficiently processes batch inference by dynamically grouping requests and overlapping computation with I/O. For offline processing of large datasets, vLLM can process multiple sequences in parallel, maximizing throughput. This capability makes vLLM suitable for both real-time serving and batch workloads like dataset annotation or bulk inference tasks.
SKILL.md
rendered from the published skill — quoted content, verbatim
vLLM - High-Performance LLM Serving
When to use
Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.
Quick start
vLLM achieves 24x higher throughput than standard transformers through PagedAttention (block-based KV cache) and continuous batching (mixing prefill/decode requests).
Installation:
pip install vllm
Basic offline inference: ```python from vllm import LLM, SamplingParams
llm = LLM(model="meta-llama/Meta-Llama-3-8B-Instruct") sampling = SamplingParams(temperature=0.7, max_tokens=256)
outputs = llm.generate(["Explain quantum computing"],
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 5 files
skills/mlops/inference/serving-llms-vllm/SKILL.md
skills/mlops/inference/serving-llms-vllm/references/optimization.md
skills/mlops/inference/serving-llms-vllm/references/quantization.md
skills/mlops/inference/serving-llms-vllm/references/server-deployment.md
skills/mlops/inference/serving-llms-vllm/references/troubleshooting.md