---
id: Prism-Shadow/penguin-harness/vllm
version: "2be01ff0"
license: Apache-2.0
install: manual
updated: 2026-07-27
---

# vllm — vLLM runs open-weight models on local GPUs with high-throughput inference behind an OpenAI-compatible endpoint, ready for chat and agent workloads. Configure context length and quantization to fit your hardware, then register the endpoint with PenguinHarness to make it available to your agents.

Publisher: Prism-Shadow · Stars: 205 · Updated: 2026-07-27

Install (manual): `git clone https://github.com/Prism-Shadow/penguin-harness`

## SKILL.md

# vLLM Serving

vLLM serves open-weight LLMs on local GPUs with high-throughput inference behind an OpenAI-compatible API, ready for chat and agent workloads.

## Before you start

If the user's message only invokes this skill (e.g. "use vllm skill") without a concrete request, ask the user what they want. Do not run any command until the goal is clear.

Ask the user which model to serve; if they have no preference, recommend the small default [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B). Also ask what context length the workload needs.

vLLM needs an NVIDIA or AMD GPU. Engine choice follows the user's preference: Ollama also runs on GPUs and is the simpler default — pick vLLM for high-throughput serving, and Ollama on macOS or CPU-only machines, which vLLM does not serve. Confirm the hardware first:

```bash
nvidia-smi          # NVIDIA: GPU model and free VRAM (AMD ROCm: rocm-smi)
python3 --version   # a recent Python is required
```

The model must fit the available VRAM — model size and context length drive the serve flags below.

## Suggested workflow

1. Ask the user which model to serve; with no preference, recommend [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B).
2. Pick the engine the user prefers: vLLM for high-throughput GPU serving; Ollama is the simple default and the choice on macOS or CPU-only machines.
3. Serve on a free port, with the tool-calling flags whenever agents will call it (see below).
4. Verify with `curl http://localhost:8000/v1/models`.
5. Register the endpoint: `penguin config model add ... --client-type openai --base-url http://localhost:8000/v1` — a served model is not visible to Penguin until added.
6. Confirm the new entry with `penguin config model list`.

## Install

Use a fresh virtual environment (or `uv`):

```bash
python3 -m venv .venv && source .venv/bin/activate
pip install vllm
```

## Serve

```bash
vllm serve Qwen/Qwen3.5-0.8B --port 8000
```

This exposes an OpenAI-compatible API at `http://localhost:8000/v1`. Key flags:

- `--served-model-name <name>` — the model id clients request (defaults to the model path).
- `--api-key <key>` — require this bearer token on every request.
- `--max-model-len <n>` — context window; agent sessions need a large one.
- `--gpu-memory-utilization <0..1>` — fraction of VRAM to claim (default 0.9).
- `--tensor-parallel-size <n>` — shard across `n` GPUs.
- `--dtype <auto|bfloat16|float16>` and `--quantization <awq|gptq|fp8>` — precision and quantized weights.

If the port is taken, pick a free one — never kill a process already listening on it.

## Tool calling — required for agents

Agent harnesses (PenguinHarness included) send `tools` with their requests. vLLM must opt in at startup:

```bash
vllm serve Qwen/Qwen3.5-0.8B --enable-auto-tool-choice --tool-call-parser hermes
```

Choose the parser for the model family — e.g. `hermes` for Qwen models, `llama3_json` for Llama models. Without these flags, requests that set tool_choice fail with `400 "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set`.

## Verify

```bash
curl http://localhost:8000/v1/models
```

## Register with PenguinHarness

Model configuration is the penguin CLI's job — `penguin config model add` registers an endpoint and `penguin config model list` shows what has been registered. A served model is not visible to Penguin until you add it:

```bash
penguin config model add --provider custom --client-type openai \
  --base-url http://localhost:8000/v1 --model-id <served-model-name> --api-key <key>
penguin config model list   # the new entry should now be listed
```

## Troubleshooting

- Out of memory at startup: lower `--gpu-memory-utilization` or `--max-model-len`, or serve a quantized model.
- Long prompts truncated or context-length errors: raise `--max-model-len` (bounded by VRAM).
- `400` on tool calls: restart the server with the tool-calling flags above.


[View on SkillFed](https://skillfed.io/Prism-Shadow/penguin-harness/vllm) · [View on GitHub](https://github.com/Prism-Shadow/penguin-harness)