serving-llms-vllm
serving-llms-vllm accelerates LLM inference for production environments through PagedAttention-based memory optimization and continuous batching. It supports OpenAI-compatible endpoints, quantization methods like GPTQ and AWQ, and tensor parallelism across multiple GPUs. Use this skill when deploying scalable LLM services that demand both low latency and high request throughput.
serving-llms-vllm enables high-throughput production LLM API deployment with optimized inference latency.
AI-generated summary based on this skill's SKILL.md
Decision gist · record as of 2026-06-16
serving-llms-vllm enables high-throughput production LLM API deployment with optimized inference latency. serving-llms-vllm accelerates LLM inference for production environments through PagedAttention-based memory optimization and continuous batching. It supports OpenAI-compatible endpoints, quantization methods like GPTQ and AWQ, and tensor parallelism across multiple GPUs. Use this skill when deploying scalable LLM services that demand both low latency and high request throughput.
Use it when
- serving-llms-vllm supports quantization methods including GPTQ and AWQ, allowing you to serve large language models on limited GPU memory.
- serving-llms-vllm optimizes inference through multiple mechanisms: PagedAttention reduces KV cache memory overhead.
Verify before relying
Read SKILL.md below before installing (5 files). Open directory: indexed for reading, not audited.
Install
Orchestra-Research/AI-Research-SKILLs/vllm · repository language: TeX
Open directory. Skills are indexed for reading, not audited. Review a skill's body before installing it.
Frequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How does serving-llms-vllm deploy LLM APIs with high throughput and low latency?
serving-llms-vllm achieves high throughput and low latency through PagedAttention-based memory optimization and continuous batching. PagedAttention reduces memory fragmentation by allocating KV cache in fixed-size blocks, enabling higher batch sizes. Continuous batching allows new requests to join the batch mid-execution without waiting for prior requests to finish, dramatically reducing time-to-first-token and improving overall request throughput in production environments.
What quantization methods does serving-llms-vllm support for serving large models?
serving-llms-vllm supports quantization methods including GPTQ and AWQ, allowing you to serve large language models on limited GPU memory. Quantization reduces model size and memory footprint by representing weights with lower precision, enabling deployment of models like Llama 70B on consumer-grade GPUs. This approach trades minimal accuracy loss for dramatic memory savings and faster inference.
How can you optimize LLM inference latency and throughput using serving-llms-vllm?
serving-llms-vllm optimizes inference through multiple mechanisms: PagedAttention reduces KV cache memory overhead, continuous batching maximizes GPU utilization by processing multiple requests concurrently, prefix caching reuses computation for repeated prompts, and speculative decoding accelerates token generation. Tensor parallelism distributes large models across multiple GPUs. Together these techniques lower per-request latency while increasing requests-per-second throughput.
Does serving-llms-vllm support tensor parallelism for multi-GPU deployment?
Yes, serving-llms-vllm supports tensor parallelism to scale LLM inference across multiple GPUs. Tensor parallelism splits model weights and computations horizontally across GPUs, enabling deployment of very large models that exceed single-GPU memory. This approach reduces per-GPU memory requirements and can improve throughput when combined with continuous batching and other optimization techniques.
What monitoring and troubleshooting capabilities does serving-llms-vllm provide?
serving-llms-vllm exposes Prometheus metrics for monitoring inference performance, including request latency, throughput, and GPU utilization. Common troubleshooting scenarios include out-of-memory errors (addressed via quantization or tensor parallelism), low throughput (improved via continuous batching tuning), and first-token latency optimization. Load testing and performance metrics help identify bottlenecks in production deployments.
Can serving-llms-vllm provide an OpenAI-compatible endpoint for LLM serving?
Yes, serving-llms-vllm provides OpenAI-compatible endpoints, allowing drop-in replacement of OpenAI APIs with local LLM inference. This compatibility simplifies migration from cloud-hosted models to self-hosted deployments while maintaining the same client code and API contracts. The endpoint supports standard chat completion and text completion requests with all serving-llms-vllm optimizations enabled.
SKILL.md
Rendered from the published skill. Quoted content, verbatim.
vLLM - High-Performance LLM Serving
Quick start
vLLM achieves 24x higher throughput than standard transformers through PagedAttention (block-based KV cache) and continuous batching (mixing prefill/decode requests).
Installation:
pip install vllm
Basic offline inference:
from vllm import LLM, SamplingParams
llm = LLM(model="meta-llama/Llama-3-8B-Instruct")
sampling = SamplingParams(temperature=0.7, max_tokens=256)
outputs = llm.generate(["Explain quantum computing"], sampling)
print(outputs[0].outputs[0].text)
OpenAI-compatible server: ```bash vllm serve
(truncated - see the full file via the links below)
File tree — 5 files
12-inference-serving/vllm/SKILL.md
12-inference-serving/vllm/references/optimization.md
12-inference-serving/vllm/references/quantization.md
12-inference-serving/vllm/references/server-deployment.md
12-inference-serving/vllm/references/troubleshooting.md
Let your AI agent find skills like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 56,283 agent skills by what they can do, searchable in plain language.
wish › “Deploy production LLM APIs with high throughput and low latency”
Give your agent the search over MCP, or paste the wish link into any chat. No install? Search from any chat →
Related skills
vllm-server guides you through deploying and configuring vLLM—a high-performance open-source LLM serving engine—for production workloads. Set up continuous batching, multi-GPU tensor parallelism, model quantization, and OpenAI-compatible API endpoints to serve models like Llama and Mistral at scale. Includes Docker deployment, performance tuning, monitoring with Prometheus metrics, and troubleshooting for common VRAM and throughput issues.
vLLM accelerates LLM inference for production deployments using PagedAttention and continuous batching to achieve significantly higher throughput than standard approaches. It provides OpenAI-compatible endpoints, supports quantization methods like AWQ and GPTQ for memory-constrained setups, and enables tensor parallelism across multiple GPUs. Deploy via Docker, monitor with Prometheus metrics, and handle batch processing or real-time serving workflows.
Model Serving equips you to run LLMs and traditional ML models in production using optimized inference engines like vLLM and BentoML. It covers GPU memory optimization, continuous batching, streaming response patterns, and integration with frontend applications for real-world AI deployments.
This skill delivers operational patterns for tuning LLM inference performance, cost, and reliability in production. It covers latency budgeting, continuous batching, KV-cache optimization, speculative decoding, quantization strategies, and parallelism choices—with decision trees and checklists to guide infrastructure and serving stack tuning.
A structured decision guide for deploying and tuning vLLM in production environments. Covers the PagedAttention memory model, continuous batching, quantization tradeoffs, tensor/pipeline parallelism choices, and step-by-step operational workflows for diagnosing throughput, latency, and out-of-memory issues. Includes comparisons to alternative inference engines and guidance on when vLLM is the right fit.
vLLM runs open-weight models on local GPUs with high-throughput inference behind an OpenAI-compatible endpoint, ready for chat and agent workloads. Configure context length and quantization to fit your hardware, then register the endpoint with PenguinHarness to make it available to your agents.
More skills tensorrt-llm (MIT) · agentsop-llm-engine-selection (MIT) · llama-cpp (MIT)