📘 AI System Design: LLMs, RAG and Agents

By 2026 AI components appear in ordinary system design interviews. The bar reported by interview-prep sources is: integrate an LLM the way you would integrate a cache or a queue. That means knowing where embeddings live, what an LLM call costs in latency and money on the hot path, and what the fallback is when the model is down. Interviewers also grade cost per request and operations (observability, rollback, on-call) explicitly.

This page gives the vocabulary, the serving architecture, and worked designs. Model names and prices change quickly, so numbers here are illustrative and the reasoning is what to reuse.

Table of Contents

1. What Is Different About AI Systems
2. LLM Concepts a System Designer Needs
3. LLM Inference Serving
4. Design: A ChatGPT-Style Assistant
5. Design: RAG Over Private Documents
6. Design: Agents and Tool Use
7. AI Gateway and Semantic Caching
8. Evaluation and Observability
9. Reliability and Cost
10. Practice Prompts
11. Questions and Answers
12. Sources

1. What Is Different About AI Systems

A useful mental model: an LLM is a slow, expensive, probabilistic, stateless function called over the network. You wrap it with caches, queues, retries, validators and observability like any flaky dependency, and you add evaluation because its correctness is statistical.

2. LLM Concepts a System Designer Needs

2.1 Tokens and Context

• Text is split into tokens (roughly 3 to 4 characters, or three quarters of an English word).
• The context window is the maximum tokens of input plus output the model can handle in one call.
• Providers bill per token, typically separate prices for input and output, with output costing several times more.
• Everything the model should know must be in the prompt: system instructions, conversation history, retrieved documents, tool results. Assembling the prompt within a token budget is a core design task.

2.2 Two Phases of Inference

Because decode is memory-bandwidth bound (each new token needs the model weights read from GPU memory), batching many requests together is the main way to raise throughput.

2.3 The KV Cache

For each token in the context, the model stores key and value vectors for every layer so it never recomputes them.

KV bytes per token = 2 (K and V) x layers x kv_heads x head_dim x bytes_per_value

Illustrative 70B-class model with grouped-query attention, fp16:
2 x 80 layers x 8 kv heads x 128 dim x 2 bytes = 327,680 bytes, about 320 KB per token
An 8,000 token conversation therefore holds about 2.6 GB of KV cache

GPU memory holds model weights plus KV caches of all in-flight requests, so KV cache memory limits concurrency. This is why long contexts are expensive and why KV cache management is the center of modern serving engines.

2.4 Model Choice Levers

Default order of attack: better prompt, then retrieval, then a better or smaller model routed per task, then fine-tuning last.

3. LLM Inference Serving

If you host models yourself (or reason about how providers do), this is the architecture.

3.1 Request Path

3.2 Techniques Behind Modern Engines

3.3 Operational Points

• Autoscale on queue depth, in-flight tokens and TTFT, not CPU. GPU nodes are scarce and slow to start because weights are tens to hundreds of GB, so keep warm capacity and cache weights on local disk.
• Multi-tenancy: rate limit in tokens per minute as well as requests, use priority classes (interactive above batch), and fair queuing so one tenant cannot starve others.
• Stream responses. Users tolerate a long answer if the first token arrives quickly. Design for client disconnects (cancel generation to free GPU).
• Batch workloads (bulk classification, document extraction) go through an offline queue on cheaper or idle capacity, often with provider batch APIs at a discount.
• Routing: cheap model for easy requests, larger model for hard ones, based on a classifier or a cascade with confidence.
• Failure: GPU node loss aborts in-flight requests. Retry idempotently from the start (generation is not resumable) or return partial output with an error marker.

4. Design: A ChatGPT-Style Assistant

4.1 Requirements and Numbers

• Multi-turn chat with streaming, history, file upload, safety filtering, model selection.
• Assume 10M DAU, 5 conversations per day, average 3 turns each.
• Latency: first token under 1.5 s (p95), then a smooth stream.

Requests = 10M x 5 x 3 = 150M turns per day = about 1,700 per second average, 5,000 peak
Tokens per turn (illustrative): 1,500 input (system prompt plus history) and 300 output
Daily tokens = 150M x 1,800 = 270B tokens per day. Token cost, not servers, is the main line item.

4.2 Architecture

4.3 Deep Dive

• Conversation storage: append-only messages keyed by (conversation_id, seq). Wide-column or a sharded SQL table. Users list conversations by recency, so index (user_id, updated_at).
• Context building: the window is finite, so decide what to include: system prompt, the last N turns, a rolling summary of older turns, retrieved chunks, tool results. Count tokens with the model's tokenizer. Put stable content first so provider prompt caching and prefix caching hit.
• Streaming: the chat service proxies SSE from the model to the client. Persist the assistant message when the stream ends (or incrementally), and handle a client that reconnects mid-stream by resuming from stored partial output.
• Safety layers: input checks (policy, PII), output checks, and system-prompt hardening. Run the moderator in parallel with generation where you can and cut the stream if it flags. Keep prompt injection in mind for uploaded files.
• Model router: send simple requests to a small fast model and hard ones to a larger model. Expose a user choice for premium tiers.
• Rate limits and quotas: per user and per tenant, in tokens per minute and per day, with a cheap pre-check using an estimate of input tokens and reservation of a max output budget, then a true-up after completion.
• Fallbacks: if the primary model or region is down, fail over to another provider or a smaller model, with a banner. Timeouts on time to first token as well as total time.
• Feedback loop: thumbs up or down, regenerate clicks, and edits are signals stored for evaluation and later tuning. Sample conversations for review with privacy controls.
• Privacy: retention policies, opt-out from training, encryption, regional storage, redaction in logs.

4.4 Trade-offs

Context length (quality) vs cost and latency, larger model vs responsiveness, aggressive caching vs personalization correctness, and safety filtering strictness vs false positives.

5. Design: RAG Over Private Documents

Retrieval-Augmented Generation fetches relevant documents at question time and gives them to the model, so answers are grounded in your data and can cite sources. Typical asks: "chat with our docs", enterprise search, customer support bots.

5.1 Two Pipelines

5.2 Ingestion Decisions

5.3 Vector Search Basics

Embeddings map text to vectors so that similar meaning gives nearby vectors. Finding the exact nearest neighbors is too slow at scale, so indexes use approximate nearest neighbor (ANN) search.

Memory for 10M chunks x 1024 dimensions x 4 bytes (float32) = about 41 GB before index overhead
int8 quantization = about 10 GB, binary = about 1.3 GB (with rescoring on full vectors on disk)

Where to store: pgvector inside Postgres for up to millions or tens of millions of vectors with transactional simplicity, dedicated engines (Qdrant, Milvus, Weaviate, Pinecone) for larger scale and richer filtering, Elasticsearch or OpenSearch when you already run them and want hybrid search in one place.

5.4 Retrieval Quality Techniques

Most RAG failures are retrieval and ingestion failures, not generation failures. Fix them in this order:

1. Evaluation set first. A few hundred real questions with expected source chunks.
2. Hybrid search: run keyword search (BM25) and vector search in parallel and fuse the rankings. Keywords catch exact terms (error codes, names, SKUs) that embeddings miss. Fusion by Reciprocal Rank Fusion is simple and robust.
3. Reranking: retrieve 30 to 100 candidates cheaply, rescore with a cross-encoder that reads query and passage together, keep the top 3 to 8.
4. Metadata filters and ACL trimming: filter by tenant and permissions inside the retrieval query, never after generation. A user must never receive text they cannot access.
5. Query rewriting: resolve pronouns from chat history, expand acronyms, decompose multi-part questions.
6. Better chunking and contextual headers.

// runnable
// Reciprocal Rank Fusion: score(d) = sum over rankings of 1 / (k + rank)
function rrf(rankings, k = 60) {
  const scores = new Map();
  for (const list of rankings) {
    list.forEach((id, i) => scores.set(id, (scores.get(id) || 0) + 1 / (k + i + 1)));
  }
  return [...scores.entries()].sort((a, b) => b[1] - a[1]).map(([id]) => id);
}

const keywordResults = ['d3', 'd1', 'd7', 'd2'];   // BM25 order
const vectorResults  = ['d1', 'd9', 'd3', 'd4'];   // embedding order
console.log(rrf([keywordResults, vectorResults])); // d1 and d3 rise: both lists like them

5.5 Generation Stage

• Instruct the model to answer only from the provided context, cite chunk ids, and say "I do not know" when the context lacks the answer.
• Pass chunks with source labels, ordered by relevance (models attend less to the middle of very long contexts).
• Validate: citations must refer to real retrieved chunks, optionally run an entailment or LLM check for unsupported claims.
• Cache: exact-match cache for repeated questions, embedding cache for queries, and prefix caching for the system prompt.

5.6 Scaling and Operations

5.7 Trade-offs

Larger chunks and more context raise recall and cost, reranking adds latency but is usually the highest-value quality upgrade, and fine-grained ACLs complicate the index. Long-context models reduce but do not remove the need for retrieval: cost, latency and attention quality still favor sending only what matters.

6. Design: Agents and Tool Use

An agent is an LLM in a loop that can call tools to act on the world: search, read files, call APIs, run code.

6.1 The Loop

6.2 Tool Interface: MCP

The Model Context Protocol (MCP) is an open standard (introduced in 2024) for connecting models to tools and data. A host application runs MCP clients that talk to MCP servers over JSON-RPC (locally over stdio, remotely over HTTP). Servers expose tools (functions the model can call), resources (data it can read) and prompts (templates). It has become a common integration layer across agent frameworks, which means a single tool server can be reused across clients, and also that a tool server is a security boundary you must review.

6.3 Design Concerns

6.4 Workflow or Agent?

Use a fixed workflow (code decides the steps, the LLM fills in specific steps) when the process is known. Use an agent (LLM decides the steps) when the path is unpredictable. Workflows are cheaper, faster, easier to test and to secure. Most production value comes from workflows with a few LLM steps.

6.5 Multi-Agent Patterns

Orchestrator with specialist workers, planner and executor, reviewer and generator. Each hop adds latency, cost and failure modes, so justify multi-agent designs with a measured quality gain.

7. AI Gateway and Semantic Caching

An AI gateway is the single entry point between your applications and model providers, like an API gateway specialized for LLMs.

Responsibilities:

• Key management so application code never holds provider keys, plus per-team and per-feature budgets.
• Routing and fallback across models and providers on errors, timeouts, and cost policy.
• Token-aware rate limiting and quota enforcement.
• Retries with backoff for 429 and 5xx (respect provider retry hints), with a circuit breaker per provider.
• Guardrails: PII redaction, prompt and output policy checks, schema validation of structured output.
• Observability and cost attribution by team, feature and user.
• Prompt management: versioned prompts, rollout and rollback without redeploying apps.

7.1 Caching Layers

Semantic caching suits FAQ-like traffic. Scope it by tenant and by user context, keep a high threshold, and avoid it for personalized or time-sensitive answers.

8. Evaluation and Observability

You cannot manage what you do not measure, and LLM output is non-deterministic.

8.1 Offline Evaluation

• Build a golden dataset of real inputs with expected outputs or grading rubrics, grown from production failures.
• Score with exact checks where possible (schema valid, tool called correctly, citation present), LLM-as-judge for open text, and human review for calibration.
• Judges are biased (favoring longer answers, their own family of model, first position). Calibrate them against human labels and randomize order.
• Run the suite in CI. Any change to a prompt, model, retrieval setting or tool should pass evals before release.

8.2 Online Evaluation and Monitoring

Release safely: shadow a new model on real traffic without showing answers, then canary it to a small percentage with automatic rollback on metric regressions.

8.3 Traces

One trace per request with spans for retrieval, rerank, each LLM call, each tool call. OpenTelemetry has emerging conventions for generative AI spans. Log prompts and outputs carefully, with redaction and retention limits, because they contain user data.

9. Reliability and Cost

9.1 Failure Modes and Responses

9.2 Cost Model

cost per request = input_tokens x input_price + output_tokens x output_price   (minus cache discounts)

Illustrative prices: 1 dollar per million input tokens, 4 dollars per million output tokens
Traffic: 1M requests per day, 2,000 input tokens, 400 output tokens each
Input  = 2B tokens per day  x 1 / 1M = 2,000 dollars per day
Output = 0.4B tokens per day x 4 / 1M = 1,600 dollars per day
Total  = 3,600 dollars per day, about 108,000 dollars per month
If 60 percent of input is a cacheable prefix billed at a 90 percent discount: input drops to about 920 dollars per day

Cost levers, roughly in order of payoff:

1. Shorter prompts (trim history, fewer and better chunks).
2. Prompt and prefix caching, semantic caching for repeated queries.
3. Route easy work to smaller models, use a cascade.
4. Limit output length and use structured output.
5. Batch offline work on discounted batch APIs.
6. Self-host open-weight models when volume is high and steady enough to keep GPUs busy (compare utilization, ops cost and staffing, not only token price).
7. Quantization and better serving efficiency (batching, prefix caching) if self-hosted.

9.3 Latency Budget Example (RAG chat, p95 target 2 s to first token)

10. Practice Prompts

Use the seven-step framework for each.

11. Questions and Answers

Q1. Why is LLM serving harder to scale than a normal web service? Requests are long, stateful (KV cache) and GPU bound, capacity is scarce and slow to add, cost is per token, and utilization depends on batching. Autoscale on queue and token metrics and use continuous batching.

Q2. What is continuous batching and why does it matter? Instead of waiting for a whole batch to finish, the scheduler swaps requests in and out at every generation step, so GPUs stay busy and short requests are not held hostage by long ones.

Q3. Your RAG bot gives wrong answers. Where do you look first? Retrieval: were the right chunks retrieved? Check chunking, parsing, hybrid search, reranking and filters. Then the prompt and grounding instructions, then the model. Add an eval set so you can measure changes.

Q4. How do you stop a user from seeing documents they should not? Store ACL metadata with each chunk, apply the user's identity as a filter inside the retrieval query (security trimming), propagate permission and delete changes to the index, and never let the model see unauthorized text.

Q5. How do you defend against prompt injection? No single fix. Treat retrieved and tool content as untrusted data, least privilege tools, confirmation for risky actions, isolate untrusted content from instructions, output and egress monitoring, and red-team tests.

Q6. Semantic cache or exact cache? Exact and provider prefix caching are safe and cheap wins. Semantic caching helps repetitive FAQ traffic and needs tenant scoping, a high threshold and exclusion of personalized queries.

Q7. How would you cut the bill by half? Measure tokens per feature first. Trim prompts, add prefix caching, route easy traffic to a smaller model, cap output, batch offline work, and consider self-hosting only if utilization justifies it.

Q8. How do you roll out a new model? Run the eval suite, shadow on live traffic, canary with automatic rollback on quality, latency and cost metrics, then ramp.

Q9. When do you fine-tune instead of using RAG? Fine-tune for style, format, or narrow skills the base model lacks. Use RAG for knowledge that changes or must be cited or access-controlled. They combine well.

Q10. What is the difference between a workflow and an agent, and which do you choose? A workflow follows steps you coded, an agent lets the model choose steps. Prefer workflows for known processes because they are cheaper, faster and safer, and use agents for open-ended tasks with strict budgets and permissions.

Q11. How do you handle the LLM provider being down? Timeouts and retries with backoff, a circuit breaker, failover to another provider or smaller model through the gateway, cached or non-AI fallback, and a status banner.

Q12. How large a context should you send? As small as achieves the quality target. Cost and latency grow with tokens and models can miss information buried in long contexts, so retrieve, rerank and trim.

12. Sources

• vLLM: Anatomy of a High-Throughput LLM Inference System for continuous batching, PagedAttention, prefix caching and chunked prefill.
• vLLM project and its documentation for serving features.
• Model Context Protocol for the tool and resource protocol.
• LLM System Design guide (System Design Handbook) and Aced (formerly Exponent) system design guide for how interview expectations have shifted toward AI and cost awareness.

Next: Frontend System Design or Reliability and Operations.