serving-llms-vllm
vLLM accelerates LLM inference for production deployments using PagedAttention and continuous batching to achieve significantly higher throughput than standard approaches. It provides OpenAI-compatible endpoints, supports quantization methods like AWQ and GPTQ for memory-constrained setups, and enables tensor parallelism across multiple GPUs. Deploy via Docker, monitor with Prometheus metrics, and handle batch processing or real-time serving workflows.
serving-llms-vllm enables production LLM API deployment with 24x higher throughput than standard transformers through PagedAttention and continuous batching.
AI-generated summary based on this skill's SKILL.md
Decision gist · record as of 2026-07-28
serving-llms-vllm enables production LLM API deployment with 24x higher throughput than standard transformers through PagedAttention and continuous batching. vLLM accelerates LLM inference for production deployments using PagedAttention and continuous batching to achieve significantly higher throughput than standard approaches. It provides OpenAI-compatible endpoints, supports quantization methods like AWQ and GPTQ for memory-constrained setups, and enables tensor parallelism across multiple GPUs. Deploy via Docker, monitor with Prometheus metrics, and handle batch processing or real-time serving workflows.
Use it when
- vLLM supports quantization methods including AWQ and GPTQ to compress models and reduce memory footprint.
- Yes, vLLM provides OpenAI-compatible endpoints out of the box.
Verify before relying
Read SKILL.md below before installing (5 files). Open directory: indexed for reading, not audited.
Install
NousResearch/hermes-agent/serving-llms-vllm · repository language: Python
Open directory. Skills are indexed for reading, not audited. Review a skill's body before installing it.
Frequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How does vLLM achieve high throughput LLM serving?
vLLM accelerates LLM inference through PagedAttention, which reduces memory fragmentation by storing attention key-value caches in non-contiguous memory blocks. Combined with continuous batching, vLLM dynamically schedules requests to maximize GPU utilization. This architecture enables significantly higher throughput than standard approaches while maintaining low latency, making it ideal for production deployments handling variable request volumes.
What quantization methods does vLLM support to reduce memory usage?
vLLM supports quantization methods including AWQ and GPTQ to compress models and reduce memory footprint. These techniques allow you to serve large language models on limited GPU memory by reducing model size while maintaining reasonable inference quality. Quantization is particularly valuable when deploying large models like 70B-parameter variants on constrained hardware.
Can vLLM be deployed with OpenAI API compatibility?
Yes, vLLM provides OpenAI-compatible endpoints out of the box. You can set up vLLM servers that expose the same API interface as OpenAI's services, allowing existing applications and clients to switch to vLLM without code changes. This compatibility simplifies integration into production systems and enables drop-in replacement scenarios.
How does vLLM tensor parallelism work across multiple GPUs?
vLLM tensor parallelism distributes model computations across multiple GPUs, splitting tensors along specific dimensions so each GPU handles a portion of the computation. This approach enables serving very large models that exceed single-GPU memory capacity. Combined with other optimization techniques, tensor parallelism helps achieve both high throughput and low latency in multi-GPU setups.
What deployment options are available for vLLM in production?
vLLM supports Docker containerization for consistent deployments and integrates with Kubernetes for orchestration at scale. You can monitor performance using Prometheus metrics to track throughput, latency, and resource utilization. These deployment patterns enable reliable, observable production LLM serving with support for load testing and performance benchmarking.
How does vLLM handle batch inference on large datasets?
vLLM's continuous batching scheduler efficiently processes batch inference by dynamically grouping requests and overlapping computation with I/O. For offline processing of large datasets, vLLM can process multiple sequences in parallel, maximizing throughput. This capability makes vLLM suitable for both real-time serving and batch workloads like dataset annotation or bulk inference tasks.
SKILL.md
Rendered from the published skill. Quoted content, verbatim.
vLLM - High-Performance LLM Serving
When to use
Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.
Quick start
vLLM achieves 24x higher throughput than standard transformers through PagedAttention (block-based KV cache) and continuous batching (mixing prefill/decode requests).
Installation:
pip install vllm
Basic offline inference: ```python from vllm import LLM, SamplingParams
llm = LLM(model="meta-llama/Meta-Llama-3-8B-Instruct") sampling = SamplingParams(temperature=0.7, max_tokens=256)
outputs = llm.generate(["Explain quantum computing"],
(truncated - see the full file via the links below)
File tree — 5 files
skills/mlops/inference/serving-llms-vllm/SKILL.md
skills/mlops/inference/serving-llms-vllm/references/optimization.md
skills/mlops/inference/serving-llms-vllm/references/quantization.md
skills/mlops/inference/serving-llms-vllm/references/server-deployment.md
skills/mlops/inference/serving-llms-vllm/references/troubleshooting.md
Let your AI agent find skills like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 56,283 agent skills by what they can do, searchable in plain language.
wish › “Deploy production LLM API with high throughput and low latency”
Give your agent the search over MCP, or paste the wish link into any chat. No install? Search from any chat →
Related skills
serving-llms-vllm accelerates LLM inference for production environments through PagedAttention-based memory optimization and continuous batching. It supports OpenAI-compatible endpoints, quantization methods like GPTQ and AWQ, and tensor parallelism across multiple GPUs. Use this skill when deploying scalable LLM services that demand both low latency and high request throughput.
vllm-server guides you through deploying and configuring vLLM—a high-performance open-source LLM serving engine—for production workloads. Set up continuous batching, multi-GPU tensor parallelism, model quantization, and OpenAI-compatible API endpoints to serve models like Llama and Mistral at scale. Includes Docker deployment, performance tuning, monitoring with Prometheus metrics, and troubleshooting for common VRAM and throughput issues.
Model Serving equips you to run LLMs and traditional ML models in production using optimized inference engines like vLLM and BentoML. It covers GPU memory optimization, continuous batching, streaming response patterns, and integration with frontend applications for real-world AI deployments.
This skill delivers operational patterns for tuning LLM inference performance, cost, and reliability in production. It covers latency budgeting, continuous batching, KV-cache optimization, speculative decoding, quantization strategies, and parallelism choices—with decision trees and checklists to guide infrastructure and serving stack tuning.
A structured decision guide for deploying and tuning vLLM in production environments. Covers the PagedAttention memory model, continuous batching, quantization tradeoffs, tensor/pipeline parallelism choices, and step-by-step operational workflows for diagnosing throughput, latency, and out-of-memory issues. Includes comparisons to alternative inference engines and guidance on when vLLM is the right fit.
TensorRT-LLM accelerates large language model inference on NVIDIA GPUs through advanced optimization techniques including quantization, in-flight batching, and multi-GPU parallelism. Achieve production-grade throughput and latency for real-time applications with support for 100+ models.
More skills vllm (Apache-2.0)