Production AI systems are distributed systems with new constraints at every layer

AI System Design is the discipline of wrapping an LLM in everything else a production system needs: inference servers, caches, load balancers, retrieval pipelines, agent loops, guardrails, and cost controls. This repository is a long-form written guide to all of it, organized as a linear reading path from GPU fundamentals through multi-agent coordination.

The guide's organizing insight is worth stating plainly: an AI system is a regular distributed system with an LLM dropped into the middle, plus a set of new constraints that change every familiar design decision. Requests are slow (sometimes minutes), output is non-deterministic, every call costs real money, and the bottleneck during decode is memory bandwidth rather than compute. Round-robin load balancing breaks because a 100-token request and a 100,000-token request are not equivalent work. Auto-scaling breaks because a 70B model can take up to ten minutes to load into GPU memory. Caching splits into at least four distinct layers: KV Cache inside a single request, prompt caching for shared prefixes across requests, semantic caching for meaning-equivalent queries, and embedding caching for repeated vectorization.

Inference engine selection gets a concrete decision tree: vLLM as the default, SGLang for high prefix-overlap workloads like RAG, TensorRT-LLM when NVIDIA lock-in is acceptable and maximum throughput is the goal. The chunked prefill and prefill-decode disaggregation sections explain why a long prompt's prefill phase can stall other users' token streams, and how separating prefill and decode workers onto different GPUs resolves the contention at the cost of a KV Cache transfer over a fast interconnect.

The RAG section is the longest and most detailed, covering document parsing (where the guide notes that 60 to 80 percent of real engineering effort lands), chunking strategies, hybrid BM25-plus-vector search, HyDE, cross-encoder reranking, ColBERT's per-token vectors, Agentic RAG, GraphRAG for multi-hop relational queries, and Vectorless RAG for structured documents where chunking destroys meaning.

The agent sections cover the five-part anatomy of an agent loop, the distinction between AI orchestration (developer-controlled flow) and true agents (LLM-controlled flow), loop engineering, graph engineering, MCP as the emerging standard for tool connectivity, and Agent Skills as a progressive-disclosure mechanism for procedure knowledge that keeps context cost near-constant across many loaded skills.

The guide doubles as interview-prep material: the back-of-the-envelope estimation section works through a 10-million daily active user scenario with explicit token, GPU, and cost arithmetic, and an eight-step framework for solving any AI system design problem closes the guide with the kind of scaffolding that forces useful discipline in real design work as much as in interviews.