The AI System-Design Framework

1. Quick anchor

Every AI system-design question is the same eight-stage pipeline wearing a different costume: data pipeline → retrieval → model routing → inference optimization → guardrails → evals → observability → cost. The single most important reframe is that the LLM call — the part candidates obsess over — is roughly 20% of the system; the other 80% is the data plumbing, retrieval quality, routing logic, safety perimeter, and measurement loop that decide whether the thing works in production. Interviewers are not testing whether you know what an embedding is; they are testing whether you treat the model as a component inside an engineered system with explicit cost, latency, and quality budgets. The strongest candidates drive the conversation through these stages, make capacity/latency/cost estimates out loud, and name the tradeoff at each fork (cache vs. rerank vs. route vs. rewrite). This lesson is the spine you hang every other system-design lesson on.

2. Why interviewers probe this

• IC4 — Can you name the full pipeline instead of collapsing "RAG" into "LLM + embeddings"? Do you know that retrieval, ranking, citation, guardrails, and evals are separate concerns? Signal: you don't skip stages, and you can sketch the offline/online split.
• IC5 — Can you budget? Given a request volume, do you produce a defensible per-request cost and latency number, identify the dominant term, and pick the right architecture for the scale (sub-100K docs vs. 10M+)? Signal: you reach for arithmetic unprompted and reason about which lever moves the bill.
• IC6 — Can you reason about the system's failure modes and organizational implications? Do you know that agentic workflows turn a cheap per-token price into an expensive per-task cost, that MTEB rank doesn't predict in-domain success, and that observability ≠ monitoring? Signal: you make scale-aware, second-order arguments and defend a position under pushback.

3. Concept build-up

3.1 The eight stages, and why the order matters

The framework is a directed pipeline because each stage constrains the next. Garbage chunking caps retrieval recall no matter how good your reranker is; weak retrieval caps answer quality no matter how large your model is; and no amount of model quality saves you if you can't measure regressions. Walking the stages in order is also a conversational tactic — it stops you from rabbit-holing on the LLM while the interviewer is waiting to hear about freshness or guardrails.

1. Data pipeline — ingestion, normalization (dedup, format standardization, metadata extraction), chunking, embedding generation. This is offline and must be physically separated from serving so a re-index doesn't starve live queries of CPU/IO.
2. Retrieval — query rewrite/expansion, BM25 + vector search, Reciprocal Rank Fusion, cross-encoder reranking. This is where most quality lives.
3. Model routing — classify request complexity, send to the cheapest sufficient model, define fallbacks.
4. Inference optimization — caching (prompt + semantic), continuous batching, quantization, speculative decoding.
5. Guardrails — input guardrails (prompt injection, jailbreak, moderation) and output guardrails (policy, hallucination self-check).
6. Evals — golden sets + LLM-as-judge, offline regression gates and online sampling.
7. Observability — per-request traces of the full chain, with cost/latency/quality attribution.
8. Cost — the running total, and the lever that decides which optimization is worth building.

3.2 The LLM is ~20% of the system

Here is the load-bearing claim. If you list the components that determine whether an AI product succeeds — data quality, chunking, embedding choice, hybrid retrieval, reranking, freshness/sync, routing, caching, guardrails, golden sets, judges, tracing, cost controls — the raw model invocation is one box among a dozen. The common pitfall the rubric explicitly calls out is "treating RAG as LLM + embeddings instead of the full pipeline." The model is also the part you have the least leverage over: you can't change its weights (usually), but you fully control the retrieval that feeds it and the evals that grade it. So the engineering effort — and the interview signal — lives in the other 80%. When an interviewer hears you say "the model is the easy part, the retrieval and the eval loop are the hard part," you've signaled seniority in one sentence.

3.3 Offline vs. online: the separation that prevents outages

The single most common architecture mistake is letting indexing compete with serving. The offline pipeline (ingest → chunk → embed → index) is throughput-bound and bursty; the online pipeline (retrieve → rank → generate) is latency-bound and must hold a p90 TTFT under ~2 seconds for enterprise RAG. Run them on shared resources and a 10M-document re-index spikes your serving latency. Separate them, and you also get clean ownership boundaries: the data team owns freshness SLAs, the serving team owns latency SLAs.

3.4 Capacity, latency, and cost estimation — the part candidates skip

This is the skill that separates IC5 from IC4. You should be able to produce a back-of-envelope estimate in under two minutes. The method:

• Capacity: requests/day → requests/second (divide by ~86,400, then multiply by a peak factor of 2–5x for diurnal spikes). 5M/day ≈ 58 rps average, ~150–290 rps peak.
• Latency: add the stage budget. A production hybrid retrieval pipeline (query expansion → BM25 → vector → RRF → rerank) adds 200–400ms; the LLM TTFT is typically the dominant single term. Semantic-cache hits return in <100ms and bypass the model entirely.
• Cost: the formula is requests × (input_tokens + output_tokens) × price_per_token, minus whatever your cache hit rate removes. The dangerous subtlety: agentic workflows make 50–200 model calls per task, so a "cheap" $0.40/M-token model becomes expensive per task. Always cost the task, not the token.

3.5 Driving the prompt: a conversational protocol

When the interviewer says "design X," do this in order: (1) clarify scale, latency SLA, freshness requirement, and quality bar — these pick your architecture; (2) sketch the eight stages as boxes; (3) estimate capacity/cost out loud; (4) deep-dive the 1–2 stages that dominate this problem (retrieval for RAG, sandboxing for agents, latency for voice); (5) name tradeoffs at each fork; (6) close with failure modes and observability. The clarifying questions are not throat-clearing — "sub-100K docs vs. 10M+" and "24-hour vs. sub-minute freshness" lead to different systems (pgvector vs. Turbopuffer; batch re-index vs. CDC).

3.6 The forks, and what changes them

The high-bar signal is trade-off clarity: knowing when to cache vs. rerank vs. route vs. rewrite.

• Cache when queries repeat (>30% semantic overlap) — cheapest possible win, but risks staleness and wrong-answer-on-near-miss if the similarity threshold (0.85–0.95) is loose.
• Rerank when recall is fine but precision is poor — a cross-encoder on the top 20→top 5 buys 1–9% recall improvement at +50–150ms.
• Route when request complexity varies — send classification/extraction to a small model (e.g. Haiku-class) for ~12x cost reduction with minimal quality loss; route hard reasoning to the large model.
• Rewrite when queries are underspecified — query expansion before retrieval, at the cost of an extra small-model call.

The same scale-awareness applies to every stage: under 10M chunks, pgvector's operational simplicity beats specialized stores; past 10M, you move to serverless ANN (Turbopuffer, HNSW/IVFPQ) for cost-effectiveness.

4. Minimal implementation

Here is a runnable skeleton that makes the eight stages concrete and the cost arithmetic explicit. It's a planning/estimation harness — the kind of thing you'd actually whiteboard, parameterized so you can change one assumption and see the bill move.

from dataclasses import dataclass
 
@dataclass
class Stage:
    name: str
    p50_ms: float          # added latency at the median
    per_call_usd: float    # marginal $ this stage adds per request
 
# The eight-stage pipeline as a list of boxes with budgets attached.
PIPELINE = [
    Stage("query_rewrite",   p50_ms=40,   per_call_usd=0.00005),  # small model
    Stage("bm25",            p50_ms=20,   per_call_usd=0.0),
    Stage("vector_search",   p50_ms=35,   per_call_usd=0.00002),  # ANN + embed
    Stage("rrf_fusion",      p50_ms=5,    per_call_usd=0.0),
    Stage("rerank",          p50_ms=120,  per_call_usd=0.0002),   # cross-encoder
    Stage("guardrail_in",    p50_ms=15,   per_call_usd=0.00003),
    Stage("llm_generate",    p50_ms=900,  per_call_usd=0.0149),   # the ~20% box
    Stage("guardrail_out",   p50_ms=15,   per_call_usd=0.00003),
]
 
def estimate(requests_per_day: int, cache_hit_rate: float):
    paid = requests_per_day * (1 - cache_hit_rate)
    # Latency: cache hits return <100ms; misses pay the full chain.
    miss_latency = sum(s.p50_ms for s in PIPELINE)
    # Cost: only the LLM + paid stages run on a miss.
    cost_per_miss = sum(s.per_call_usd for s in PIPELINE)
    daily_cost = paid * cost_per_miss
    llm_share = sum(s.per_call_usd for s in PIPELINE if s.name == "llm_generate")
    return {
        "p50_miss_latency_ms": miss_latency,
        "cache_hit_latency_ms": 90,
        "paid_calls_per_day": int(paid),
        "daily_cost_usd": round(daily_cost, 2),
        "monthly_cost_usd": round(daily_cost * 30, 2),
        "llm_pct_of_cost": round(100 * llm_share / cost_per_miss, 1),
    }
 
print(estimate(requests_per_day=5_000_000, cache_hit_rate=0.6))
# -> p50 miss latency ~1150ms, ~2M paid calls/day,
#    ~$29.8K/day, ~$895K/month, LLM ~96% of *marginal $ per call*

Two things this surfaces that interviewers reward. First, the LLM box is ~20% of the stages but ~96% of the marginal dollars per call — "the LLM is 20% of the system" is a claim about engineering surface area and failure modes, not about cost share. State that distinction explicitly; conflating them is a trap. Second, changing one assumption (cache_hit_rate) is the highest-leverage edit in the file — which is exactly why caching is the first cost lever in real systems. Swap requests_per_day and cache_hit_rate to pressure-test any scenario the interviewer throws at you.

5. Production tradeoffs

The cross-cutting story: stacking model-, system-, and application-level optimizations reaches ~80% cost reduction vs. naive serving. Model-level (quantization FP16→INT8/INT4: 2–4x memory, 95–99% accuracy retained), system-level (continuous batching: 3–10x throughput; PagedAttention: up to 24x; speculative decoding: 2–3x, but only worthwhile when decode is memory-bound, and a badly chosen k can raise cost 175%), and application-level (prompt caching: 80–90% latency cut on cached prefixes; semantic caching: 68.8% API-call reduction; routing: 2–5x aggregate savings). At scale the dominant failure mode shifts from "wrong answer" to "expensive task" — because agentic loops make 50–200 calls each, the per-token price war ($30/M in 2023 → $0.40/M now) is offset by call volume per task. Cost the task.

6. How it's asked

7. Pitfalls & flashcards

• Collapsing the pipeline. "RAG = LLM + embeddings" ignores retrieval, ranking, confidence, citation, guardrails, evals, and observability — the rubric's #1 pitfall.
• No arithmetic. Designing without a cost/latency estimate is an automatic IC5 fail. Always divide requests/day, add the stage latency budget, and multiply tokens × price.
• Trusting MTEB rank. Top-3 MTEB models ranked 5th/7th/2nd on real legal retrieval; the winner was 11th on MTEB. Benchmark is a useful prior, not a decision oracle — test in-domain.
• Confusing monitoring with observability. Monitoring tracks known metrics (latency, error rate); observability captures per-request traces so you can explore unknown failures. You need both.
• Pricing the token, not the task. Agentic loops make 50–200 calls/task — cost the task.
• Sharing offline and online resources. A re-index that starves live queries is a self-inflicted outage; separate the pipelines.
• Blind speculative decoding. Worthwhile only when decode is memory-bound; a bad k can raise cost 175%.

Flashcard. Drive the eight stages — data → retrieval → routing → inference opt → guardrails → evals → observability → cost — estimate the bill out loud, and remember the LLM is ~20% of the system but ~96% of the marginal dollars.

8. Further reading

• Redis — RAG at Scale: offline/online split, hybrid retrieval latency budgets, semantic caching cost math.
• Redis — LLMOps Guide 2026: model routing, batching, unified infrastructure.
• Morph — LLM Inference Optimization: the three-layer optimization stack and ~80% aggregate cost reduction.
• Dynamic Model Routing Survey (arXiv 2603.04445): routing mechanisms and cost-quality optimization.
• Langfuse — LLM Observability: observability vs. monitoring, full-chain tracing.
• Confident AI — LLM Evaluation Playbook: golden sets and regression gates.

Next: /rag — apply this framework to the canonical "design a RAG chatbot over 10M documents" problem, where retrieval is the stage that dominates.