serving-llms-vllm
serving-llms-vllm accelerates LLM inference for production environments through PagedAttention-based memory optimization and continuous batching. It supports OpenAI-compatible endpoints, quantization methods like GPTQ and AWQ, and tensor parallelism across multiple GPUs. Use this skill when deploying scalable LLM services that demand both low latency and high request throughput.
serving-llms-vllm enables high-throughput production LLM API deployment with optimized inference latency.
AI-generated summary based on this skill's SKILL.md
Install
Orchestra-Research/AI-Research-SKILLs/vllm · repository language: TeX
git clone https://github.com/Orchestra-Research/AI-Research-SKILLs
cp -r AI-Research-SKILLs/12-inference-serving/vllm ~/.claude/skills/vllmnpx skillfed install Orchestra-Research/AI-Research-SKILLs/vllmFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How does serving-llms-vllm deploy LLM APIs with high throughput and low latency?
serving-llms-vllm achieves high throughput and low latency through PagedAttention-based memory optimization and continuous batching. PagedAttention reduces memory fragmentation by allocating KV cache in fixed-size blocks, enabling higher batch sizes. Continuous batching allows new requests to join the batch mid-execution without waiting for prior requests to finish, dramatically reducing time-to-first-token and improving overall request throughput in production environments.
What quantization methods does serving-llms-vllm support for serving large models?
serving-llms-vllm supports quantization methods including GPTQ and AWQ, allowing you to serve large language models on limited GPU memory. Quantization reduces model size and memory footprint by representing weights with lower precision, enabling deployment of models like Llama 70B on consumer-grade GPUs. This approach trades minimal accuracy loss for dramatic memory savings and faster inference.
How can you optimize LLM inference latency and throughput using serving-llms-vllm?
serving-llms-vllm optimizes inference through multiple mechanisms: PagedAttention reduces KV cache memory overhead, continuous batching maximizes GPU utilization by processing multiple requests concurrently, prefix caching reuses computation for repeated prompts, and speculative decoding accelerates token generation. Tensor parallelism distributes large models across multiple GPUs. Together these techniques lower per-request latency while increasing requests-per-second throughput.
Does serving-llms-vllm support tensor parallelism for multi-GPU deployment?
Yes, serving-llms-vllm supports tensor parallelism to scale LLM inference across multiple GPUs. Tensor parallelism splits model weights and computations horizontally across GPUs, enabling deployment of very large models that exceed single-GPU memory. This approach reduces per-GPU memory requirements and can improve throughput when combined with continuous batching and other optimization techniques.
What monitoring and troubleshooting capabilities does serving-llms-vllm provide?
serving-llms-vllm exposes Prometheus metrics for monitoring inference performance, including request latency, throughput, and GPU utilization. Common troubleshooting scenarios include out-of-memory errors (addressed via quantization or tensor parallelism), low throughput (improved via continuous batching tuning), and first-token latency optimization. Load testing and performance metrics help identify bottlenecks in production deployments.
Can serving-llms-vllm provide an OpenAI-compatible endpoint for LLM serving?
Yes, serving-llms-vllm provides OpenAI-compatible endpoints, allowing drop-in replacement of OpenAI APIs with local LLM inference. This compatibility simplifies migration from cloud-hosted models to self-hosted deployments while maintaining the same client code and API contracts. The endpoint supports standard chat completion and text completion requests with all serving-llms-vllm optimizations enabled.
SKILL.md
rendered from the published skill — quoted content, verbatim
vLLM - High-Performance LLM Serving
Quick start
vLLM achieves 24x higher throughput than standard transformers through PagedAttention (block-based KV cache) and continuous batching (mixing prefill/decode requests).
Installation:
pip install vllm
Basic offline inference:
from vllm import LLM, SamplingParams
llm = LLM(model="meta-llama/Llama-3-8B-Instruct")
sampling = SamplingParams(temperature=0.7, max_tokens=256)
outputs = llm.generate(["Explain quantum computing"], sampling)
print(outputs[0].outputs[0].text)
OpenAI-compatible server: ```bash vllm serve
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 5 files
12-inference-serving/vllm/SKILL.md
12-inference-serving/vllm/references/optimization.md
12-inference-serving/vllm/references/quantization.md
12-inference-serving/vllm/references/server-deployment.md
12-inference-serving/vllm/references/troubleshooting.md