vllm
vLLM runs open-weight models on local GPUs with high-throughput inference behind an OpenAI-compatible endpoint, ready for chat and agent workloads. Configure context length and quantization to fit your hardware, then register the endpoint with PenguinHarness to make it available to your agents.
vLLM deploys open-weight models on local GPUs with an OpenAI-compatible API for high-throughput inference.
AI-generated summary based on this skill's SKILL.md
Decision gist · record as of 2026-07-27
vLLM deploys open-weight models on local GPUs with an OpenAI-compatible API for high-throughput inference. vLLM runs open-weight models on local GPUs with high-throughput inference behind an OpenAI-compatible endpoint, ready for chat and agent workloads. Configure context length and quantization to fit your hardware, then register the endpoint with PenguinHarness to make it available to your agents.
Use it when
- vLLM provides a drop-in OpenAI-compatible API server.
- vLLM deploys open-weight models directly to GPU for fast inference.
Verify before relying
Read SKILL.md below before installing (2 files). Open directory: indexed for reading, not audited.
Install
Prism-Shadow/penguin-harness/vllm · repository language: TypeScript
Open directory. Skills are indexed for reading, not audited. Review a skill's body before installing it.
Frequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I serve open source models locally with vLLM?
vLLM runs open-weight models on your GPU with high-throughput inference. Install vLLM, specify your model (e.g., `vllm serve meta-llama/Llama-2-7b-hf`), and it exposes an OpenAI-compatible API endpoint. Configure GPU memory and context length to match your hardware, then point clients to the local endpoint.
What is vLLM's OpenAI-compatible API setup?
vLLM provides a drop-in OpenAI-compatible API server. Start the server with your chosen model, and it listens on a configurable port (default 8000). Clients can use standard OpenAI SDK calls against `http://localhost:8000/v1`, making vLLM compatible with existing agent frameworks and chat applications without code changes.
How do I deploy an LLM on GPU with vLLM?
vLLM deploys open-weight models directly to GPU for fast inference. Install vLLM, ensure CUDA is available, then run `vllm serve <model-name>`. vLLM automatically manages GPU memory, batching, and KV-cache optimization. Adjust `--gpu-memory-utilization` and `--tensor-parallel-size` for your GPU count and memory constraints.
How does vLLM enable tool calling for agent workloads?
vLLM supports tool calling for agent frameworks through structured output and function definitions. Enable auto tool choice with model-specific settings, define your tools in the request, and vLLM returns structured tool calls. Register your vLLM endpoint with PenguinHarness to make tools available to your agents seamlessly.
How do I optimize GPU memory and throughput with vLLM?
vLLM optimizes GPU memory through quantization (int8, int4), context window tuning, and batching. Use `--gpu-memory-utilization` to control memory headroom, `--tensor-parallel-size` for multi-GPU sharding, and `--max-model-len` to limit context. These settings balance throughput and memory usage for your hardware.
How do I register a vLLM endpoint with PenguinHarness?
vLLM provides an OpenAI-compatible endpoint that PenguinHarness can consume directly. In your PenguinHarness config, register the vLLM server URL (e.g., `http://localhost:8000/v1`) as a model provider. PenguinHarness will route agent requests to vLLM, enabling local inference with tool calling and chat capabilities.
SKILL.md
Rendered from the published skill. Quoted content, verbatim.
vLLM Serving
vLLM serves open-weight LLMs on local GPUs with high-throughput inference behind an OpenAI-compatible API, ready for chat and agent workloads.
Before you start
If the user's message only invokes this skill (e.g. "use vllm skill") without a concrete request, ask the user what they want. Do not run any command until the goal is clear.
Ask the user which model to serve; if they have no preference, recommend the small default Qwen/Qwen3.5-0.8B. Also ask what context length the workload needs.
vLLM needs an NVIDIA or AMD GPU. Engine choice follows the user's preference: Ollama also runs on GPUs and is the simpler default — pick vLLM for high-throughput serving, and Ollama on macOS or CPU-only machines, which vLLM does not serve. Confirm the hardware first:
```bash nvidia-smi # NVIDIA: GPU model and free VRAM (AMD ROCm:
(truncated - see the full file via the links below)
File tree — 2 files
packages/skills/skills/vllm/SKILL.md
packages/skills/skills/vllm/icon.svg
Let your AI agent find skills like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 56,283 agent skills by what they can do, searchable in plain language.
wish › “Deploy and serve open-weight LLMs locally on GPU with OpenAI-compatible API”
Give your agent the search over MCP, or paste the wish link into any chat. No install? Search from any chat →
Related skills
vllm-server guides you through deploying and configuring vLLM—a high-performance open-source LLM serving engine—for production workloads. Set up continuous batching, multi-GPU tensor parallelism, model quantization, and OpenAI-compatible API endpoints to serve models like Llama and Mistral at scale. Includes Docker deployment, performance tuning, monitoring with Prometheus metrics, and troubleshooting for common VRAM and throughput issues.
Ollama lets you deploy and serve open-weight language models on your machine with automatic GPU detection. It exposes an OpenAI-compatible API endpoint, integrates with PenguinHarness for model registration, and supports flexible context window configuration for agent workloads.
LlamaFactory streamlines LLM fine-tuning through YAML-driven configs, supporting LoRA, QLoRA, and full-parameter training methods like SFT and DPO. Register datasets, train your model, merge adapters, and export for standalone deployment or API serving. The skill guides you through GPU requirements, base model selection, and inference setup.
vLLM accelerates LLM inference for production deployments using PagedAttention and continuous batching to achieve significantly higher throughput than standard approaches. It provides OpenAI-compatible endpoints, supports quantization methods like AWQ and GPTQ for memory-constrained setups, and enables tensor parallelism across multiple GPUs. Deploy via Docker, monitor with Prometheus metrics, and handle batch processing or real-time serving workflows.
serving-llms-vllm accelerates LLM inference for production environments through PagedAttention-based memory optimization and continuous batching. It supports OpenAI-compatible endpoints, quantization methods like GPTQ and AWQ, and tensor parallelism across multiple GPUs. Use this skill when deploying scalable LLM services that demand both low latency and high request throughput.
Model Serving equips you to run LLMs and traditional ML models in production using optimized inference engines like vLLM and BentoML. It covers GPU memory optimization, continuous batching, streaming response patterns, and integration with frontend applications for real-world AI deployments.
More skills agentsop-vllm (MIT)