model-serving
Model Serving equips you to run LLMs and traditional ML models in production using optimized inference engines like vLLM and BentoML. It covers GPU memory optimization, continuous batching, streaming response patterns, and integration with frontend applications for real-world AI deployments.
Model Serving deploys self-hosted LLMs for production inference with optimized throughput and latency.
AI-generated summary based on this skill's SKILL.md
Install
ancoleman/ai-design-components/model-serving · repository language: Python
git clone https://github.com/ancoleman/ai-design-components
cp -r ai-design-components/skills/model-serving ~/.claude/skills/model-servingnpx skillfed install ancoleman/ai-design-components/model-servingFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I deploy LLM models in production?
Model Serving provides frameworks and best practices for deploying LLMs in production environments. You'll learn to use optimized inference engines like vLLM and BentoML that handle GPU memory optimization, continuous batching, and efficient token generation. The skill covers containerization, scaling strategies, and integration patterns to ensure your models run reliably at scale with minimal latency.
What's the difference between vLLM and Ollama for model serving?
Model Serving compares inference engines like vLLM and Ollama to help you choose the right tool. vLLM excels at high-throughput production deployments with advanced features like PagedAttention and continuous batching, while Ollama prioritizes ease of local setup. The skill guides you through their trade-offs in performance, resource requirements, and use-case fit.
How can I build AI APIs with streaming responses?
Model Serving teaches you to construct AI APIs that deliver real-time token generation using streaming patterns like Server-Sent Events (SSE). You'll learn how to integrate streaming backends with frontend applications, handle concurrent requests efficiently, and maintain low latency while delivering responses incrementally to users.
How do I implement RAG pipelines with vector databases?
Model Serving covers orchestrating Retrieval-Augmented Generation (RAG) systems by combining LLM inference with vector database integration. You'll learn to set up retrieval workflows using tools like LangChain, manage embeddings, and coordinate data flow between your serving infrastructure and knowledge bases for context-aware AI responses.
What optimization techniques improve inference latency and throughput?
Model Serving addresses GPU utilization optimization through techniques like quantization, PagedAttention memory management, and continuous batching. You'll discover how to benchmark performance, profile bottlenecks, and tune parameters to reduce latency while maximizing throughput for your specific hardware and model configuration.
Can Model Serving help with Kubernetes deployment for LLMs?
Model Serving includes guidance on deploying LLMs at scale using Kubernetes and containerized inference engines. You'll learn orchestration patterns, resource allocation strategies, and multi-model serving setups that enable reliable, scalable production deployments across distributed infrastructure.
SKILL.md
rendered from the published skill — quoted content, verbatim
Model Serving
Purpose
Deploy LLM and ML models for production inference with optimized serving engines, streaming response patterns, and orchestration frameworks. Focuses on self-hosted model serving, GPU optimization, and integration with frontend applications.
When to Use
- Deploying LLMs for production (self-hosted Llama, Mistral, Qwen)
- Building AI APIs with streaming responses
- Serving traditional ML models (scikit-learn, XGBoost, PyTorch)
- Implementing RAG pipelines with vector databases
- Optimizing inference throughput and latency
- Integrating LLM serving with frontend chat interfaces
Model Serving Selection
LLM Serving Engines
vLLM (Recommended Primary) - PagedAttention memory management (20-30x throughput improvement) - Continuous batching for dynamic request handling - OpenAI-compatible
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 15 files
skills/model-serving/SKILL.md
skills/model-serving/examples/k8s-vllm-deployment/README.md
skills/model-serving/examples/langchain-agents/README.md
skills/model-serving/examples/langchain-agents/main.py
skills/model-serving/examples/langchain-agents/requirements.txt
skills/model-serving/examples/langchain-rag-qdrant/README.md
skills/model-serving/examples/ollama-local/README.md
skills/model-serving/examples/ollama-local/main.py
skills/model-serving/examples/ollama-local/requirements.txt
skills/model-serving/examples/vllm-serving/README.md
skills/model-serving/examples/vllm-serving/main.py
skills/model-serving/examples/vllm-serving/requirements.txt
skills/model-serving/outputs.yaml
skills/model-serving/references/bentoml.md
skills/model-serving/references/inference-optimization.md