model-serving
Model Serving equips you to run LLMs and traditional ML models in production using optimized inference engines like vLLM and BentoML. It covers GPU memory optimization, continuous batching, streaming response patterns, and integration with frontend applications for real-world AI deployments.
Model Serving deploys self-hosted LLMs for production inference with optimized throughput and latency.
AI-generated summary based on this skill's SKILL.md
Decision gist · record as of 2025-12-11
Model Serving deploys self-hosted LLMs for production inference with optimized throughput and latency. Model Serving equips you to run LLMs and traditional ML models in production using optimized inference engines like vLLM and BentoML. It covers GPU memory optimization, continuous batching, streaming response patterns, and integration with frontend applications for real-world AI deployments.
Use it when
- Model Serving compares inference engines like vLLM and Ollama to help you choose the right tool.
- Model Serving teaches you to construct AI APIs that deliver real-time token generation using streaming patterns like Server-Sent Events.
Verify before relying
Read SKILL.md below before installing (21 files). Open directory: indexed for reading, not audited.
Install
ancoleman/ai-design-components/model-serving · repository language: Python
Open directory. Skills are indexed for reading, not audited. Review a skill's body before installing it.
Frequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I deploy LLM models in production?
Model Serving provides frameworks and best practices for deploying LLMs in production environments. You'll learn to use optimized inference engines like vLLM and BentoML that handle GPU memory optimization, continuous batching, and efficient token generation. The skill covers containerization, scaling strategies, and integration patterns to ensure your models run reliably at scale with minimal latency.
What's the difference between vLLM and Ollama for model serving?
Model Serving compares inference engines like vLLM and Ollama to help you choose the right tool. vLLM excels at high-throughput production deployments with advanced features like PagedAttention and continuous batching, while Ollama prioritizes ease of local setup. The skill guides you through their trade-offs in performance, resource requirements, and use-case fit.
How can I build AI APIs with streaming responses?
Model Serving teaches you to construct AI APIs that deliver real-time token generation using streaming patterns like Server-Sent Events (SSE). You'll learn how to integrate streaming backends with frontend applications, handle concurrent requests efficiently, and maintain low latency while delivering responses incrementally to users.
How do I implement RAG pipelines with vector databases?
Model Serving covers orchestrating Retrieval-Augmented Generation (RAG) systems by combining LLM inference with vector database integration. You'll learn to set up retrieval workflows using tools like LangChain, manage embeddings, and coordinate data flow between your serving infrastructure and knowledge bases for context-aware AI responses.
What optimization techniques improve inference latency and throughput?
Model Serving addresses GPU utilization optimization through techniques like quantization, PagedAttention memory management, and continuous batching. You'll discover how to benchmark performance, profile bottlenecks, and tune parameters to reduce latency while maximizing throughput for your specific hardware and model configuration.
Can Model Serving help with Kubernetes deployment for LLMs?
Model Serving includes guidance on deploying LLMs at scale using Kubernetes and containerized inference engines. You'll learn orchestration patterns, resource allocation strategies, and multi-model serving setups that enable reliable, scalable production deployments across distributed infrastructure.
SKILL.md
Rendered from the published skill. Quoted content, verbatim.
Model Serving
Purpose
Deploy LLM and ML models for production inference with optimized serving engines, streaming response patterns, and orchestration frameworks. Focuses on self-hosted model serving, GPU optimization, and integration with frontend applications.
When to Use
- Deploying LLMs for production (self-hosted Llama, Mistral, Qwen)
- Building AI APIs with streaming responses
- Serving traditional ML models (scikit-learn, XGBoost, PyTorch)
- Implementing RAG pipelines with vector databases
- Optimizing inference throughput and latency
- Integrating LLM serving with frontend chat interfaces
Model Serving Selection
LLM Serving Engines
vLLM (Recommended Primary) - PagedAttention memory management (20-30x throughput improvement) - Continuous batching for dynamic request handling - OpenAI-compatible
(truncated - see the full file via the links below)
File tree — 15 files
skills/model-serving/SKILL.md
skills/model-serving/examples/k8s-vllm-deployment/README.md
skills/model-serving/examples/langchain-agents/README.md
skills/model-serving/examples/langchain-agents/main.py
skills/model-serving/examples/langchain-agents/requirements.txt
skills/model-serving/examples/langchain-rag-qdrant/README.md
skills/model-serving/examples/ollama-local/README.md
skills/model-serving/examples/ollama-local/main.py
skills/model-serving/examples/ollama-local/requirements.txt
skills/model-serving/examples/vllm-serving/README.md
skills/model-serving/examples/vllm-serving/main.py
skills/model-serving/examples/vllm-serving/requirements.txt
skills/model-serving/outputs.yaml
skills/model-serving/references/bentoml.md
skills/model-serving/references/inference-optimization.md
Let your AI agent find skills like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 56,283 agent skills by what they can do, searchable in plain language.
wish › “Deploy self-hosted LLM for production inference with optimized serving”
Give your agent the search over MCP, or paste the wish link into any chat. No install? Search from any chat →
Related skills
vllm-server guides you through deploying and configuring vLLM—a high-performance open-source LLM serving engine—for production workloads. Set up continuous batching, multi-GPU tensor parallelism, model quantization, and OpenAI-compatible API endpoints to serve models like Llama and Mistral at scale. Includes Docker deployment, performance tuning, monitoring with Prometheus metrics, and troubleshooting for common VRAM and throughput issues.
serving-llms-vllm accelerates LLM inference for production environments through PagedAttention-based memory optimization and continuous batching. It supports OpenAI-compatible endpoints, quantization methods like GPTQ and AWQ, and tensor parallelism across multiple GPUs. Use this skill when deploying scalable LLM services that demand both low latency and high request throughput.
vLLM accelerates LLM inference for production deployments using PagedAttention and continuous batching to achieve significantly higher throughput than standard approaches. It provides OpenAI-compatible endpoints, supports quantization methods like AWQ and GPTQ for memory-constrained setups, and enables tensor parallelism across multiple GPUs. Deploy via Docker, monitor with Prometheus metrics, and handle batch processing or real-time serving workflows.
A structured decision guide for deploying and tuning vLLM in production environments. Covers the PagedAttention memory model, continuous batching, quantization tradeoffs, tensor/pipeline parallelism choices, and step-by-step operational workflows for diagnosing throughput, latency, and out-of-memory issues. Includes comparisons to alternative inference engines and guidance on when vLLM is the right fit.
This skill enables dynamic scaling of LLM inference workloads across Kubernetes clusters using KEDA and Prometheus metrics tied to GPU utilization and request queues. It covers vLLM deployment, queue-based job scaling with Redis, spot instance strategies, and cluster autoscaler configuration to handle traffic spikes while optimizing costs.
Run production ML inference on Kubernetes using KServe or NVIDIA Triton, with built-in support for canary traffic splitting, request-based autoscaling, and GPU resource allocation. The skill covers model versioning, A/B testing patterns, and dynamic batching for throughput optimization.
More skills Llm Inference (unlicensed) · vllm (Apache-2.0) · Ml Engineering (unlicensed) · tensorrt-llm (MIT) · huggingface-local-models (Apache-2.0) · testing-llm (MIT) · agentsop-llm-engine-selection (MIT)