vllm
vLLM runs open-weight models on local GPUs with high-throughput inference behind an OpenAI-compatible endpoint, ready for chat and agent workloads. Configure context length and quantization to fit your hardware, then register the endpoint with PenguinHarness to make it available to your agents.
vLLM deploys open-weight models on local GPUs with an OpenAI-compatible API for high-throughput inference.
AI-generated summary based on this skill's SKILL.md
Install
Prism-Shadow/penguin-harness/vllm · repository language: TypeScript
git clone https://github.com/Prism-Shadow/penguin-harness
cp -r penguin-harness/packages/skills/skills/vllm ~/.claude/skills/vllmnpx skillfed install Prism-Shadow/penguin-harness/vllmFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I serve open source models locally with vLLM?
vLLM runs open-weight models on your GPU with high-throughput inference. Install vLLM, specify your model (e.g., `vllm serve meta-llama/Llama-2-7b-hf`), and it exposes an OpenAI-compatible API endpoint. Configure GPU memory and context length to match your hardware, then point clients to the local endpoint.
What is vLLM's OpenAI-compatible API setup?
vLLM provides a drop-in OpenAI-compatible API server. Start the server with your chosen model, and it listens on a configurable port (default 8000). Clients can use standard OpenAI SDK calls against `http://localhost:8000/v1`, making vLLM compatible with existing agent frameworks and chat applications without code changes.
How do I deploy an LLM on GPU with vLLM?
vLLM deploys open-weight models directly to GPU for fast inference. Install vLLM, ensure CUDA is available, then run `vllm serve <model-name>`. vLLM automatically manages GPU memory, batching, and KV-cache optimization. Adjust `--gpu-memory-utilization` and `--tensor-parallel-size` for your GPU count and memory constraints.
How does vLLM enable tool calling for agent workloads?
vLLM supports tool calling for agent frameworks through structured output and function definitions. Enable auto tool choice with model-specific settings, define your tools in the request, and vLLM returns structured tool calls. Register your vLLM endpoint with PenguinHarness to make tools available to your agents seamlessly.
How do I optimize GPU memory and throughput with vLLM?
vLLM optimizes GPU memory through quantization (int8, int4), context window tuning, and batching. Use `--gpu-memory-utilization` to control memory headroom, `--tensor-parallel-size` for multi-GPU sharding, and `--max-model-len` to limit context. These settings balance throughput and memory usage for your hardware.
How do I register a vLLM endpoint with PenguinHarness?
vLLM provides an OpenAI-compatible endpoint that PenguinHarness can consume directly. In your PenguinHarness config, register the vLLM server URL (e.g., `http://localhost:8000/v1`) as a model provider. PenguinHarness will route agent requests to vLLM, enabling local inference with tool calling and chat capabilities.
SKILL.md
rendered from the published skill — quoted content, verbatim
vLLM Serving
vLLM serves open-weight LLMs on local GPUs with high-throughput inference behind an OpenAI-compatible API, ready for chat and agent workloads.
Before you start
If the user's message only invokes this skill (e.g. "use vllm skill") without a concrete request, ask the user what they want. Do not run any command until the goal is clear.
Ask the user which model to serve; if they have no preference, recommend the small default Qwen/Qwen3.5-0.8B. Also ask what context length the workload needs.
vLLM needs an NVIDIA or AMD GPU. Engine choice follows the user's preference: Ollama also runs on GPUs and is the simpler default — pick vLLM for high-throughput serving, and Ollama on macOS or CPU-only machines, which vLLM does not serve. Confirm the hardware first:
```bash nvidia-smi # NVIDIA: GPU model and free VRAM (AMD ROCm:
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 2 files
packages/skills/skills/vllm/SKILL.md
packages/skills/skills/vllm/icon.svg