{"enrichment":{"faq":[{"a":"ai-llm-inference addresses latency optimization through multiple layers: continuous batching to maximize GPU utilization, KV-cache optimization with techniques like PagedAttention to reduce memory pressure, speculative decoding to parallelize token generation, and quantization (FP8, INT8) to reduce model size and bandwidth. The skill provides latency budgeting frameworks, P99 tail-latency measurement patterns, and decision trees for choosing between vLLM, TensorRT-LLM, and other serving stacks based on your workload's throughput and latency SLO requirements.","q":"How to optimize LLM inference latency in production?"},{"a":"ai-llm-inference guides you through measured quality validation for quantization choices. It covers FP8 and INT8 strategies with trade-offs: INT8 post-training quantization for broad compatibility, FP8 for newer hardware with better numerical stability, and mixed-precision approaches when certain layers need higher precision. The skill includes benchmarking checklists to validate perplexity, token accuracy, and inference speed before production deployment, helping you balance cost reduction against quality loss.","q":"What quantization strategies should I use for my LLM?"},{"a":"ai-llm-inference provides comparative guidance on serving stacks. vLLM excels at continuous batching and PagedAttention-based memory efficiency with broad model support and lower operational overhead. TensorRT-LLM offers tighter GPU optimization, better multi-GPU scaling via tensor and pipeline parallelism, and superior throughput on NVIDIA hardware. The skill includes decision criteria: choose vLLM for rapid iteration and model diversity; choose TensorRT-LLM when maximizing throughput or deploying on constrained GPU clusters.","q":"How does vLLM vs TensorRT-LLM compare for serving?"},{"a":"ai-llm-inference combines multiple cost-reduction levers: quantization to shrink model size and reduce memory bandwidth costs, model tiering and routing to direct simple queries to smaller models, prefix caching and response caching to amortize compute across similar requests, and speculative decoding with draft models to reduce latency and thus batch size. The skill provides cost-per-token formulas, capacity-planning worksheets, and checklists for profiling and benchmarking workloads under realistic load to identify your highest-impact optimization.","q":"What's the best way to reduce LLM inference cost per token?"},{"a":"ai-llm-inference covers HA patterns including multi-region failover, load balancing across serving instances, circuit breakers and graceful degradation, and SLO-driven scaling policies. It addresses continuous batching trade-offs (higher latency variance under load), replica placement for fault tolerance, and monitoring for tail latency and error rates. The skill includes decision trees for choosing between stateless request routing and stateful session affinity, and guidance on capacity planning to meet P99 latency budgets during traffic spikes.","q":"How do I design a high-availability LLM serving architecture?"},{"a":"ai-llm-inference includes patterns for resource-constrained environments: quantization to INT8 or GGUF format for llama.cpp on CPU, KV-cache pruning and context-window tuning for memory-limited devices, and speculative decoding with tiny draft models to maintain acceptable latency. While edge deployment is a lower-weight intent, the skill provides checklists for profiling and benchmarking on target hardware, helping you validate that quantized or pruned models meet latency and quality requirements before production rollout.","q":"Can ai-llm-inference help with edge or CPU deployment?"}],"shadow_tags":["performance-tuning","cost-reduction","production-serving","model-compression","distributed-inference","throughput-optimization","latency-critical","resource-constrained","hardware-acceleration","reliability-patterns"],"summary_rewrite":"This skill delivers operational patterns for tuning LLM inference performance, cost, and reliability in production. It covers latency budgeting, continuous batching, KV-cache optimization, speculative decoding, quantization strategies, and parallelism choices\u2014with decision trees and checklists to guide infrastructure and serving stack tuning."},"files":[{"bytes":16665,"path":"frameworks/shared-skills/skills/ai-llm-inference/SKILL.md","sha256":"87387b5146628d7b6a2c84d867c862a6d1a4e4ce6570bcf445b89761e71ff6db","url":"https://skillfed.io/files/vasilyu1983/AI-Agents-public/ai-llm-inference/d269ad1b/SKILL.md"}],"id":"vasilyu1983/AI-Agents-public/ai-llm-inference","links":{"html":"https://skillfed.io/vasilyu1983/AI-Agents-public/ai-llm-inference","md":"https://skillfed.io/vasilyu1983/AI-Agents-public/ai-llm-inference.md","repo":"https://github.com/vasilyu1983/AI-Agents-public"},"meta":{"agents_supported":[],"first_seen":"2026-07-28","forks":17,"language":"Python","last_updated":"2026-07-13","license":"MIT","name":"ai-llm-inference","publisher":"vasilyu1983","stars":69},"relations":{"similar":[{"id":"agentsope/SkillAlchemy/agentsop-vllm"},{"id":"synthetic-sciences/openscience/vllm"},{"id":"agentsope/SkillAlchemy/agentsop-llm-engine-selection"},{"id":"Orchestra-Research/AI-Research-SKILLs/vllm"},{"id":"NousResearch/hermes-agent/serving-llms-vllm"},{"id":"OpenLAIR/dr-claw/vllm"},{"id":"synthetic-sciences/openscience/tensorrt-llm"},{"id":"Orchestra-Research/AI-Research-SKILLs/tensorrt-llm"},{"id":"moltis-org/moltis/serving-llms-vllm"},{"id":"graniet/kheish/vllm"}]},"slug":{"owner":"vasilyu1983","repo":"AI-Agents-public","skill":"ai-llm-inference"},"version":"d269ad1b"}