Serving LLMs on your own GPUs

Ask an agent to serve a model on your own GPUs and it will hand you a command in seconds. The command will start. It will pass a smoke test. Then it falls over: the context window was set to the model's architectural maximum instead of the longest request anyone actually sends, so the KV cache has room for almost nothing; the model was sharded across cards with no NVLink between them; the autoscaler is watching CPU while the request queue grows. None of that is exotic. It is what general knowledge about serving looks like when nobody wrote down the conditions.

The more useful thing that comes out of reading these documents is that the tool is rarely the hard part. Two skills can sit under different publisher accounts, declare the same name and the same author in their own front matter, and still disagree about which command-line flags exist — one of them teaching a flag the other says was removed. So the reader's real question is not "vLLM or llama.cpp". It is: which document is this, when was it written, and does it say what has to be true for its advice to work?

Top picks

skill	publisher	license	verdict	repo updated
agentsop-llm-engine-selection	agentsope	MIT	Install before you pick an engine	2026-06-30
agentsop-vllm	agentsope	MIT	Install once you have picked vLLM	2026-06-30
vllm	Prism-Shadow	Apache-2.0	Install if your agent calls tools	2026-07-27
huggingface-local-models	huggingface	Apache-2.0	Install if you serve GGUF locally	2026-07-27
llm-inference-scaling	BagelHole	MIT	Install if you run on Kubernetes	2026-05-22

That last column is the repository's record date, not a date any of these documents claims for itself. The two can disagree, and it is the second one you want.

Engine choice: settle the constraints before the benchmark

agentsop-llm-engine-selection from agentsope makes the agent eliminate before it optimises. No NVIDIA hardware drops TensorRT-LLM; an open-source-only policy drops it again on licence grounds; weights that exist as GGUF push you to llama.cpp or Ollama regardless of what any benchmark says. Only then does it map workload to engine, and then it forces a topology check: tensor parallelism issues an all-reduce after every layer, NVLink carries that at roughly 900 GB/s, and PCIe Gen4 at roughly 32 GB/s is about 28× slower.

That check is the pick's whole value. Its verdict on the PCIe-only cluster is that switching engines will not help, because "The engine isn't the variable; the interconnect is." Compare model-serving from ancoleman, whose engine choice is a short decision tree offering a bare "2-8x faster than vLLM" — and whose front matter is a name and a description, nothing else. By its repository record it is the oldest document named on this page, and that record is the only date it has — the document claims none for itself. Caveat on the pick: it stamps dated: 2026-05 on itself and opens by telling you to re-verify before you commit a quarter's engineering budget to anything it recommends.

Tuning vLLM: take the guide that orders the fixes

agentsop-vllm, also from agentsope, is a triage document rather than a flag list. Out-of-memory at startup has a fixed sequence: drop the context length to the realistic prompt-plus-output, then --gpu-memory-utilization, then --kv-cache-dtype fp8, then --max-num-seqs, and only then raise the tensor-parallel degree. It is equally clear about what does not help — prefix caching accelerates prefill, so if outputs are long and prefixes never repeat, the gain is roughly zero and you have spent KV memory for it.

It and vllm-server from BagelHole disagree outright on --enforce-eager. BagelHole offers it as the way to minimise latency for interactive use, annotated as disabling CUDA graph capture. agentsop-vllm calls it a debugging and memory workaround that makes decode slower, and tells you to fix the underlying memory problem instead. That is the pick. It also draws its own edges: a closing section names single-user laptop work, CPU and edge machines, architectures vLLM has not covered yet, and structured generation tighter than its guided decoding as cases where vLLM is the wrong tool entirely. Caveat: it reasons more than it copy-pastes — there is one runnable serve command, at the very end.

Serving for an agent: the two flags that decide whether tool calls work

vllm from Prism-Shadow is short and operational. It refuses to run anything until the goal is clear, checks the hardware with nvidia-smi first, and asks what context length the workload needs before writing any flags. It is also plain about where it stops: vLLM needs an NVIDIA or AMD GPU, and on macOS or a CPU-only machine the skill hands the job to Ollama, which it calls the simpler default anyway. Then it names a failure an agent setup can hit the moment it sends a tool-calling request: without --enable-auto-tool-choice and --tool-call-parser, a request that sets tool_choice comes back as a 400 whose message says which flags are missing.

It also tells you how to pick between the parsers — "Choose the parser for the model family" — and then names two, hermes for Qwen and llama3_json for Llama. BagelHole's vllm-server gives you one structured-output snippet with llama3_json wired into it, against a Llama 3.1 model. That pairing is correct as written; what is missing is any rule for what to change when the model is not a Llama, so the snippet copies cleanly into a setup where it silently no longer applies. Caveat: the closing steps register the endpoint with the penguin CLI. Skip those unless you run that harness.

Running GGUF locally: get the exact file, not the family

huggingface-local-models solves one narrow problem properly: which file do you actually download. Its workflow is URL-first — search the Hub filtered to llama.cpp apps, open the repo's local-app page, prefer the quant that page marks compatible, then confirm exact .gguf filenames through the model tree API before launching llama-server. It keeps repo-native labels rather than normalising them, and it warns that mmproj-*.gguf files are projector weights, not the checkpoint.

llama-cpp from Orchestra-Research puts more on the page itself: a GGUF format table with Q4_K_M as the recommended default and Q2_K marked testing only, and the make flags for Metal, CUDA and ROCm. Set the two SKILL.md files side by side and Orchestra-Research's looks broader. It isn't: huggingface-local-models keeps the same material one level down, in bundled reference files it points at by name — one for format tables and quality tradeoffs, one for Metal, CUDA, ROCm and CPU builds. What its own page spends its space on is the step the other one skips: confirming the exact filename before you download it, plus the --hf-repo and --hf-file fallback for repos whose naming does not follow the convention. Caveat: huggingface-local-models declares no licence in its own front matter.

Autoscaling: scale on queue depth and cache pressure, not CPU

llm-inference-scaling from BagelHole wires the autoscaler to signals a language model actually produces. Its KEDA object watches two vLLM Prometheus series — requests waiting, threshold 10, and average KV cache occupancy, threshold 0.8 — with a 300-second cooldown before any scale-down, and the deployment beneath it holds its readiness probe off for the first 60 seconds of a pod's life. It keeps the minimum replica count at 1 rather than scaling to zero, and pre-pulls weights into a shared volume, which it credits with a 5–10× faster pod start.

k8s-autoscaling from rohitg00 is the better general Kubernetes reference — HPA, VPA, KEDA triggers, scale-to-zero, and the ordering rule that resource requests must exist before HPA can compute utilisation at all. But its worked examples target CPU utilisation at 70% and an HTTP request-rate query, and both sit flat while a GPU replica queues. Caveat: the BagelHole skill assumes KEDA, the NVIDIA GPU operator and a Prometheus scrape are already in place.

How do you tell two same-named serving skills apart?

Skill names are not namespaced. A skill called serving-llms-vllm can be either of two different documents, and the publisher account in the address is not the author — both of these name Orchestra Research as author in their own front matter.

The version lines differ, and so do the commands. serving-llms-vllm from Orchestra-Research declares version 1.0.0; it starts the server with --enable-metrics and --metrics-port 9090, then scrapes metrics from port 9090, and reaches for speculative decoding with --speculative-model. serving-llms-vllm hosted under NousResearch declares version 1.0.1; it drops both metrics flags and says Prometheus metrics are exposed automatically at /metrics on the API port, and it notes that --speculative-model was removed in favour of --speculative-config, which takes JSON. An agent handed the 1.0.0 text will write a serve command around two flags that no longer do what it thinks.

The same shape shows up around llama.cpp. Llama Cpp from graniet declares the same author and the same version as Orchestra-Research's llama-cpp, with a Kheish tool-name mapping preamble on the front and the name field dropped from its front matter.

So the check is mechanical: open the body, read the version line, and diff its commands against the tool's current flags before trusting any of them. The publisher and the body are a skill's identity; the name is only its topic.

What to check before you install one of these

The licence lives in two places, and they can disagree. The two are the skill's own front matter and the record for the repository that carries it. rohitg00's k8s-autoscaling declares Apache-2.0 in its front matter while the repository record says MIT. graniet's Llama Cpp does it the other way round: MIT in the front matter, Apache-2.0 on the repository record. Neither is a defect, and neither is a reason to skip the skill. It does mean that if redistribution matters to you, the front-matter line is a claim to verify against the repository, not an answer.

A speed multiplier with no workload attached is a hypothesis. tensorrt-llm from Orchestra-Research advertises 24,000 tokens/sec for Llama 3-8B on an H100 and 100× over PyTorch. ancoleman's model-serving puts TensorRT-LLM at 2-8x faster than vLLM, and vLLM itself at a 20-30x throughput improvement. agentsop-llm-engine-selection, citing published comparisons, puts the same TensorRT-LLM gap at 30–50% in saturated NVIDIA-only deployments. Neither of the first two attaches a concurrency level or a prompt-length distribution to its number, and without those the numbers are not comparable to each other, let alone to your traffic. ai-llm-inference from vasilyu1983 writes the discipline into the skill itself: performance ratios are hypotheses until a source is cited and the workload is comparable.

Read the date the document claims for itself. A repository can be updated weekly while the serving guidance inside it was written against a release that has since renamed a flag.

What you have now avoided

The opening failure was a serving config that starts, passes a smoke test, and then behaves badly under real traffic. The test that separates a skill worth installing from one worth skipping is whether the document states the conditions under which its own advice holds — hardware, workload, version — rather than handing over a number or a flag with nothing attached.

All five picks state them. agentsop-llm-engine-selection makes the constraint set an explicit opening step and dates itself. agentsop-vllm keys each fix to a symptom and marks its own boundaries, including where vLLM is the wrong tool. Prism-Shadow's vllm states its hardware requirement and points at Ollama for macOS and CPU-only machines. huggingface-local-models ties its quant choice to what the repo page marks compatible and to the memory you have. llm-inference-scaling names the metric, the threshold and the prerequisite behind each scaling rule.

Install them and the three failures that opened this are closed: a context window set to the model's architectural maximum instead of the longest request anyone sends, a shard spread across cards that have no NVLink between them, and an autoscaler watching a signal a GPU replica never moves. What is left is a serving stack whose settings each have a reason you can point at.