{"enrichment":{"faq":[{"a":"serving-llms-vllm achieves high throughput and low latency through PagedAttention-based memory optimization and continuous batching. PagedAttention reduces memory fragmentation by allocating KV cache in fixed-size blocks, enabling higher batch sizes. Continuous batching allows new requests to join the batch mid-execution without waiting for prior requests to finish, dramatically reducing time-to-first-token and improving overall request throughput in production environments.","q":"How does serving-llms-vllm deploy LLM APIs with high throughput and low latency?"},{"a":"serving-llms-vllm supports quantization methods including GPTQ and AWQ, allowing you to serve large language models on limited GPU memory. Quantization reduces model size and memory footprint by representing weights with lower precision, enabling deployment of models like Llama 70B on consumer-grade GPUs. This approach trades minimal accuracy loss for dramatic memory savings and faster inference.","q":"What quantization methods does serving-llms-vllm support for serving large models?"},{"a":"serving-llms-vllm optimizes inference through multiple mechanisms: PagedAttention reduces KV cache memory overhead, continuous batching maximizes GPU utilization by processing multiple requests concurrently, prefix caching reuses computation for repeated prompts, and speculative decoding accelerates token generation. Tensor parallelism distributes large models across multiple GPUs. Together these techniques lower per-request latency while increasing requests-per-second throughput.","q":"How can you optimize LLM inference latency and throughput using serving-llms-vllm?"},{"a":"Yes, serving-llms-vllm supports tensor parallelism to scale LLM inference across multiple GPUs. Tensor parallelism splits model weights and computations horizontally across GPUs, enabling deployment of very large models that exceed single-GPU memory. This approach reduces per-GPU memory requirements and can improve throughput when combined with continuous batching and other optimization techniques.","q":"Does serving-llms-vllm support tensor parallelism for multi-GPU deployment?"},{"a":"serving-llms-vllm exposes Prometheus metrics for monitoring inference performance, including request latency, throughput, and GPU utilization. Common troubleshooting scenarios include out-of-memory errors (addressed via quantization or tensor parallelism), low throughput (improved via continuous batching tuning), and first-token latency optimization. Load testing and performance metrics help identify bottlenecks in production deployments.","q":"What monitoring and troubleshooting capabilities does serving-llms-vllm provide?"},{"a":"Yes, serving-llms-vllm provides OpenAI-compatible endpoints, allowing drop-in replacement of OpenAI APIs with local LLM inference. This compatibility simplifies migration from cloud-hosted models to self-hosted deployments while maintaining the same client code and API contracts. The endpoint supports standard chat completion and text completion requests with all serving-llms-vllm optimizations enabled.","q":"Can serving-llms-vllm provide an OpenAI-compatible endpoint for LLM serving?"}],"shadow_tags":["api-serving","gpu-optimization","model-quantization","distributed-inference","production-deployment","latency-optimization","memory-efficiency","batch-processing","performance-monitoring"],"summary_rewrite":"serving-llms-vllm accelerates LLM inference for production environments through PagedAttention-based memory optimization and continuous batching. It supports OpenAI-compatible endpoints, quantization methods like GPTQ and AWQ, and tensor parallelism across multiple GPUs. Use this skill when deploying scalable LLM services that demand both low latency and high request throughput."},"files":[{"bytes":9026,"path":"12-inference-serving/vllm/SKILL.md","sha256":"57eaf97e982fe758b6fd0ef2281b6961940e4a1c8258f303b0ce02d55d0636c6","url":"https://skillfed.io/files/Orchestra-Research/AI-Research-SKILLs/vllm/bb643a79/SKILL.md"}],"id":"Orchestra-Research/AI-Research-SKILLs/vllm","links":{"html":"https://skillfed.io/Orchestra-Research/AI-Research-SKILLs/vllm","md":"https://skillfed.io/Orchestra-Research/AI-Research-SKILLs/vllm.md","repo":"https://github.com/Orchestra-Research/AI-Research-SKILLs"},"meta":{"agents_supported":[],"first_seen":"2026-07-28","forks":818,"language":"TeX","last_updated":"2026-06-16","license":"MIT","name":"serving-llms-vllm","publisher":"Orchestra-Research","stars":11165},"relations":{"similar":[{"id":"OpenLAIR/dr-claw/vllm"},{"id":"synthetic-sciences/openscience/vllm"},{"id":"moltis-org/moltis/serving-llms-vllm"},{"id":"graniet/kheish/vllm"},{"id":"NousResearch/hermes-agent/serving-llms-vllm"},{"id":"BagelHole/DevOps-Security-Agent-Skills/vllm-server"},{"id":"ancoleman/ai-design-components/model-serving"},{"id":"agentsope/SkillAlchemy/agentsop-vllm"},{"id":"synthetic-sciences/openscience/tensorrt-llm"},{"id":"Orchestra-Research/AI-Research-SKILLs/tensorrt-llm"}]},"slug":{"owner":"Orchestra-Research","repo":"AI-Research-SKILLs","skill":"vllm"},"version":"bb643a79"}