ai-llm-inference
This skill delivers operational patterns for tuning LLM inference performance, cost, and reliability in production. It covers latency budgeting, continuous batching, KV-cache optimization, speculative decoding, quantization strategies, and parallelism choices—with decision trees and checklists to guide infrastructure and serving stack tuning.
ai-llm-inference optimizes LLM serving latency, throughput, and cost through batching, caching, and quantization strategies.
AI-generated summary based on this skill's SKILL.md
Decision gist · record as of 2026-07-13
ai-llm-inference optimizes LLM serving latency, throughput, and cost through batching, caching, and quantization strategies. This skill delivers operational patterns for tuning LLM inference performance, cost, and reliability in production. It covers latency budgeting, continuous batching, KV-cache optimization, speculative decoding, quantization strategies, and parallelism choices—with decision trees and checklists to guide infrastructure and serving stack tuning.
Use it when
- ai-llm-inference guides you through measured quality validation for quantization choices.
- ai-llm-inference provides comparative guidance on serving stacks.
Verify before relying
Read SKILL.md below before installing (30 files). Open directory: indexed for reading, not audited.
Install
vasilyu1983/AI-Agents-public/ai-llm-inference · repository language: Python
Open directory. Skills are indexed for reading, not audited. Review a skill's body before installing it.
Frequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How to optimize LLM inference latency in production?
ai-llm-inference addresses latency optimization through multiple layers: continuous batching to maximize GPU utilization, KV-cache optimization with techniques like PagedAttention to reduce memory pressure, speculative decoding to parallelize token generation, and quantization (FP8, INT8) to reduce model size and bandwidth. The skill provides latency budgeting frameworks, P99 tail-latency measurement patterns, and decision trees for choosing between vLLM, TensorRT-LLM, and other serving stacks based on your workload's throughput and latency SLO requirements.
What quantization strategies should I use for my LLM?
ai-llm-inference guides you through measured quality validation for quantization choices. It covers FP8 and INT8 strategies with trade-offs: INT8 post-training quantization for broad compatibility, FP8 for newer hardware with better numerical stability, and mixed-precision approaches when certain layers need higher precision. The skill includes benchmarking checklists to validate perplexity, token accuracy, and inference speed before production deployment, helping you balance cost reduction against quality loss.
How does vLLM vs TensorRT-LLM compare for serving?
ai-llm-inference provides comparative guidance on serving stacks. vLLM excels at continuous batching and PagedAttention-based memory efficiency with broad model support and lower operational overhead. TensorRT-LLM offers tighter GPU optimization, better multi-GPU scaling via tensor and pipeline parallelism, and superior throughput on NVIDIA hardware. The skill includes decision criteria: choose vLLM for rapid iteration and model diversity; choose TensorRT-LLM when maximizing throughput or deploying on constrained GPU clusters.
What's the best way to reduce LLM inference cost per token?
ai-llm-inference combines multiple cost-reduction levers: quantization to shrink model size and reduce memory bandwidth costs, model tiering and routing to direct simple queries to smaller models, prefix caching and response caching to amortize compute across similar requests, and speculative decoding with draft models to reduce latency and thus batch size. The skill provides cost-per-token formulas, capacity-planning worksheets, and checklists for profiling and benchmarking workloads under realistic load to identify your highest-impact optimization.
How do I design a high-availability LLM serving architecture?
ai-llm-inference covers HA patterns including multi-region failover, load balancing across serving instances, circuit breakers and graceful degradation, and SLO-driven scaling policies. It addresses continuous batching trade-offs (higher latency variance under load), replica placement for fault tolerance, and monitoring for tail latency and error rates. The skill includes decision trees for choosing between stateless request routing and stateful session affinity, and guidance on capacity planning to meet P99 latency budgets during traffic spikes.
Can ai-llm-inference help with edge or CPU deployment?
ai-llm-inference includes patterns for resource-constrained environments: quantization to INT8 or GGUF format for llama.cpp on CPU, KV-cache pruning and context-window tuning for memory-limited devices, and speculative decoding with tiny draft models to maintain acceptable latency. While edge deployment is a lower-weight intent, the skill provides checklists for profiling and benchmarking on target hardware, helping you validate that quantized or pruned models meet latency and quality requirements before production rollout.
SKILL.md
Rendered from the published skill. Quoted content, verbatim.
LLMOps - Inference & Optimization - Production Skill Hub
Modern Best Practices (January 2026):
- Treat inference as a systems problem: SLOs, tail latency, retries, overload, and cache strategy.
- Use continuous batching / smart scheduling when serving many concurrent requests (Orca scheduling: https://www.usenix.org/conference/osdi22/presentation/yu).
- Use KV-cache aware serving (PagedAttention/vLLM: https://arxiv.org/abs/2309.06180) and efficient attention kernels (FlashAttention: https://arxiv.org/abs/2205.14135).
- Use speculative decoding when latency is critical and draft-model quality is acceptable (speculative decoding: https://arxiv.org/abs/2302.01318).
- Quantize only with measured quality impact and rollback plan (quantization must be validated on your eval set).
This skill provides production-ready operational patterns for optimizing LLM inference performance, cost, and reliability. It centralizes decision rules, optimization strategies,
(truncated - see the full file via the links below)
File tree — 15 files
frameworks/shared-skills/skills/ai-llm-inference/SKILL.md
frameworks/shared-skills/skills/ai-llm-inference/assets/batching/template-batching-config.md
frameworks/shared-skills/skills/ai-llm-inference/assets/benchmarking/template-latency-throughput-test.md
frameworks/shared-skills/skills/ai-llm-inference/assets/caching/template-prefix-caching.md
frameworks/shared-skills/skills/ai-llm-inference/assets/checklists/inference-review-checklist.md
frameworks/shared-skills/skills/ai-llm-inference/assets/inference/template-deepspeed-inference.md
frameworks/shared-skills/skills/ai-llm-inference/assets/inference/template-tensorrtllm-config.md
frameworks/shared-skills/skills/ai-llm-inference/assets/inference/template-vllm-config.md
frameworks/shared-skills/skills/ai-llm-inference/assets/quantization/template-awq.md
frameworks/shared-skills/skills/ai-llm-inference/assets/quantization/template-gguf.md
frameworks/shared-skills/skills/ai-llm-inference/assets/quantization/template-gptq.md
frameworks/shared-skills/skills/ai-llm-inference/assets/serving/template-high-throughput-setup.md
frameworks/shared-skills/skills/ai-llm-inference/assets/serving/template-llm-api.md
frameworks/shared-skills/skills/ai-llm-inference/data/sources.json
frameworks/shared-skills/skills/ai-llm-inference/references/batching-and-scheduling.md
Let your AI agent find skills like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 56,283 agent skills by what they can do, searchable in plain language.
wish › “Optimize LLM inference latency, throughput, or cost for production workloads”
Give your agent the search over MCP, or paste the wish link into any chat. No install? Search from any chat →
Related skills
A structured decision guide for deploying and tuning vLLM in production environments. Covers the PagedAttention memory model, continuous batching, quantization tradeoffs, tensor/pipeline parallelism choices, and step-by-step operational workflows for diagnosing throughput, latency, and out-of-memory issues. Includes comparisons to alternative inference engines and guidance on when vLLM is the right fit.
This skill guides you through selecting an LLM serving engine by analyzing hardware topology, workload profile, and operational constraints rather than benchmark headlines. It maps seven engines—vLLM, SGLang, TensorRT-LLM, TGI, llama.cpp, Ollama, and MLX—to their strengths across GPU clusters, edge devices, and single-user scenarios, then walks you through a five-step decision workflow to eliminate incompatible options and benchmark your top candidates.
serving-llms-vllm accelerates LLM inference for production environments through PagedAttention-based memory optimization and continuous batching. It supports OpenAI-compatible endpoints, quantization methods like GPTQ and AWQ, and tensor parallelism across multiple GPUs. Use this skill when deploying scalable LLM services that demand both low latency and high request throughput.
vLLM accelerates LLM inference for production deployments using PagedAttention and continuous batching to achieve significantly higher throughput than standard approaches. It provides OpenAI-compatible endpoints, supports quantization methods like AWQ and GPTQ for memory-constrained setups, and enables tensor parallelism across multiple GPUs. Deploy via Docker, monitor with Prometheus metrics, and handle batch processing or real-time serving workflows.
TensorRT-LLM accelerates large language model inference on NVIDIA GPUs through advanced optimization techniques including quantization, in-flight batching, and multi-GPU parallelism. Achieve production-grade throughput and latency for real-time applications with support for 100+ models.
testing-llm equips you with patterns for mocking LLM API calls, evaluating output quality across multiple dimensions, and automating test workflows through planner, generator, and healer agents. Use it to build deterministic CI tests, validate RAG pipelines, and apply structured output validation with Pydantic schemas.
More skills Llm Inference (unlicensed)