{"enrichment":{"faq":[{"a":"vLLM runs open-weight models on your GPU with high-throughput inference. Install vLLM, specify your model (e.g., `vllm serve meta-llama/Llama-2-7b-hf`), and it exposes an OpenAI-compatible API endpoint. Configure GPU memory and context length to match your hardware, then point clients to the local endpoint.","q":"How do I serve open source models locally with vLLM?"},{"a":"vLLM provides a drop-in OpenAI-compatible API server. Start the server with your chosen model, and it listens on a configurable port (default 8000). Clients can use standard OpenAI SDK calls against `http://localhost:8000/v1`, making vLLM compatible with existing agent frameworks and chat applications without code changes.","q":"What is vLLM's OpenAI-compatible API setup?"},{"a":"vLLM deploys open-weight models directly to GPU for fast inference. Install vLLM, ensure CUDA is available, then run `vllm serve <model-name>`. vLLM automatically manages GPU memory, batching, and KV-cache optimization. Adjust `--gpu-memory-utilization` and `--tensor-parallel-size` for your GPU count and memory constraints.","q":"How do I deploy an LLM on GPU with vLLM?"},{"a":"vLLM supports tool calling for agent frameworks through structured output and function definitions. Enable auto tool choice with model-specific settings, define your tools in the request, and vLLM returns structured tool calls. Register your vLLM endpoint with PenguinHarness to make tools available to your agents seamlessly.","q":"How does vLLM enable tool calling for agent workloads?"},{"a":"vLLM optimizes GPU memory through quantization (int8, int4), context window tuning, and batching. Use `--gpu-memory-utilization` to control memory headroom, `--tensor-parallel-size` for multi-GPU sharding, and `--max-model-len` to limit context. These settings balance throughput and memory usage for your hardware.","q":"How do I optimize GPU memory and throughput with vLLM?"},{"a":"vLLM provides an OpenAI-compatible endpoint that PenguinHarness can consume directly. In your PenguinHarness config, register the vLLM server URL (e.g., `http://localhost:8000/v1`) as a model provider. PenguinHarness will route agent requests to vLLM, enabling local inference with tool calling and chat capabilities.","q":"How do I register a vLLM endpoint with PenguinHarness?"}],"shadow_tags":["local-llm-serving","gpu-inference","openai-compatible-api","agent-tool-calling","model-quantization","throughput-optimization","multi-gpu-sharding","vram-management","context-window-tuning","model-registration"],"summary_rewrite":"vLLM runs open-weight models on local GPUs with high-throughput inference behind an OpenAI-compatible endpoint, ready for chat and agent workloads. Configure context length and quantization to fit your hardware, then register the endpoint with PenguinHarness to make it available to your agents."},"files":[{"bytes":4268,"path":"packages/skills/skills/vllm/SKILL.md","sha256":"f00f443ec8fac4427a91fb048ef3c7bf92a4b69c34f155195df087a6847aeaaf","url":"https://skillfed.io/files/Prism-Shadow/penguin-harness/vllm/2be01ff0/SKILL.md"}],"id":"Prism-Shadow/penguin-harness/vllm","links":{"html":"https://skillfed.io/Prism-Shadow/penguin-harness/vllm","md":"https://skillfed.io/Prism-Shadow/penguin-harness/vllm.md","repo":"https://github.com/Prism-Shadow/penguin-harness"},"meta":{"agents_supported":[],"first_seen":"2026-07-28","forks":24,"language":"TypeScript","last_updated":"2026-07-27","license":"Apache-2.0","name":"vllm","publisher":"Prism-Shadow","stars":205},"relations":{"similar":[{"id":"BagelHole/DevOps-Security-Agent-Skills/vllm-server"},{"id":"Prism-Shadow/penguin-harness/ollama"},{"id":"NousResearch/hermes-agent/serving-llms-vllm"},{"id":"moltis-org/moltis/serving-llms-vllm"},{"id":"Orchestra-Research/AI-Research-SKILLs/vllm"},{"id":"OpenLAIR/dr-claw/vllm"},{"id":"synthetic-sciences/openscience/vllm"},{"id":"graniet/kheish/vllm"},{"id":"ancoleman/ai-design-components/model-serving"},{"id":"agentsope/SkillAlchemy/agentsop-vllm"}]},"slug":{"owner":"Prism-Shadow","repo":"penguin-harness","skill":"vllm"},"version":"2be01ff0"}