The canonical AI system-design question, fully worked — from ingestion and chunking to vector-store choice, hybrid retrieval, freshness, access control, and the cost model that decides which one wins.
RAG over 10M documents is not "an LLM with embeddings" — it is two pipelines that share an index. The offline pipeline turns documents into chunks, embeddings, and metadata; the online pipeline turns a query into a retrieved-and-reranked context that an LLM answers from. The whole system lives or dies on three orthogonal axes — freshness, latency, cost — and your job in the interview is to make the tradeoffs among them explicit for this corpus, not in the abstract. The single highest-signal move is to separate indexing from serving so a re-index never competes with live queries, and to treat retrieval quality (recall, precision-by-document-type, citation correctness) as a measured metric rather than a hope. Get the chunking, the hybrid-retrieval-plus-rerank stack, and the cost model right, and everything else is operational hardening.
The words first.
1/(k+rank) per list.Step by step.
Remember this: Retrieval quality is a metric you measure, not a property you assume — everything downstream inherits its errors.
Documents are chunked, embedded and indexed offline; at query time the system retrieves, fuses, reranks, and grounds the answer. Watch the data flow.
Start by converting "10M documents" into the numbers that actually drive architecture. Assume an average document of ~3,000 tokens. At a 400-token chunk with ~15% overlap, the effective stride is ~340 tokens, so each document yields roughly 3000 / 340 ≈ 9 chunks. That is ~90M chunks. This single derivation reframes the problem: you are not building a 10M-row store, you are building a ~100M-vector store, which is squarely in "specialized ANN at scale" territory, not "toy pgvector table."
Now the index memory. A 1024-dim embedding in float32 is 1024 × 4 = 4,096 bytes ≈ 4 KB per vector. Raw vectors alone: 90M × 4 KB ≈ 360 GB. An HNSW graph adds connection overhead — budget roughly 1.5–2× the raw vector size for the graph links, so plan on ~600 GB of in-memory index for full float32 HNSW. That number is the whole reason the rest of section 3.2 exists: it forces either sharding across many nodes, quantization of the vectors (int8 cuts it ~4×), or an on-object-storage architecture that keeps cost down by not holding everything in RAM.
Name the symbols: D = documents (10,000,000). t = avg tokens/doc (3,000). c = chunk size (400 tokens). o = overlap fraction (0.15), so stride s = c × (1 − o) = 340. dim = embedding dimension (1,024). b = bytes/number (4 for float32).
t / s = 3000 / 340 ≈ 8.8 → round to 9.N = D × 9 = 90,000,000.dim × b = 1024 × 4 = 4,096 ≈ 4 KB.N × 4 KB = 90,000,000 × 4,096 ≈ 360 GB.≈ 610 GB resident memory.b = 1): raw drops to ≈ 90 GB, index ≈ 150 GB.
What it did to the data: it turned a vague "10M docs" into a concrete ~90M-vector, ~600 GB (or ~150 GB quantized) index — which is the number that decides pgvector vs. a sharded/object-storage store.The offline pipeline is normalize → chunk → embed → write. Normalization matters more than candidates expect: dedup near-identical documents (otherwise you waste embedding spend and pollute retrieval with redundant near-duplicates), standardize formats (PDF/HTML/Markdown → clean text), and extract metadata up front — source_id, tenant_id, acl_groups, version, updated_at. That metadata is load-bearing for freshness and access control later; bolting it on after indexing means a full re-index.
For chunk size, the 2026 baseline from cross-model studies (a NAACL 2025 evaluation across 48 models) is 256–512 tokens for factoid/lookup queries and up to 1,024 tokens for complex analytical queries. Use recursive splitting that respects structure (headings, paragraphs, sentences) with 10–20% overlap so a concept split across a boundary still appears whole in at least one chunk. Avoid 128-token chunks: they split mid-concept, strand context, and measurably increase hallucination because the LLM gets fragments. Page-level chunking is a strong, simple default across heterogeneous document types when you can't tune per query class.
For embeddings, the MTEB leaderboard (as of March 2026: Cohere embed-v4 ~65.2, OpenAI text-embedding-3-large ~64.6, BGE-M3 ~63.0) is a useful prior, not an oracle. The honest caveat to say out loud: in a real legal-contract retrieval study, the MTEB top-3 ranked 5th/7th/2nd in-domain, and the actual winner ranked 11th on MTEB. So: shortlist with MTEB, then run your own in-domain eval (section 3.6) on your corpus before committing. Picking an embedding model from a leaderboard alone is a classic IC4-level mistake.
This is where scale-awareness shows. There is no universal answer; there's a threshold.
The decision rule to articulate: below the threshold, optimize for operational simplicity; above it, optimize for cost-per-vector and the ability to push metadata filters (tenant, ACL, freshness) into the ANN search itself so you never retrieve a chunk the user isn't allowed to see. A store that filters after ANN retrieval will silently degrade recall when filters are selective — you ask for 10, the ANN returns 10, the filter drops 8, and you're left with 2.
Vector-only search misses exact-match queries: error codes, SKUs, statute numbers, "ISO 27001." Embeddings smear those into a neighborhood; BM25 nails them. So run both in parallel: BM25 top-20 and vector top-20. Merge with Reciprocal Rank Fusion — for each candidate, score Σ 1/(k + rank_i) across the lists it appears in (k≈60 is the usual constant), which needs no score normalization and is robust to the two systems' incomparable score scales. Then rerank the fused top candidates with a cross-encoder and keep the final top 5–10.
The payoff and the price, stated plainly: hybrid + rerank buys roughly 1–9% recall improvement over vector-only (larger on exact-match-heavy corpora), at a cost of ~200–400ms added latency for the full Query-Expansion → BM25 → Vector → RRF → Rerank pipeline. The reranker is the latency-dominant stage; cap its candidate set (e.g. rerank top-30, not top-200) and you control the cost. The tradeoff to name: rerank when precision@k matters (legal, support deflection, anything cited to users); skip it when you're feeding 20 chunks to a long-context model that can tolerate noise and latency is sacred.
Semantic caching is the biggest single cost lever on read-heavy workloads. Embed the incoming query, vector-search a cache of prior (query → answer) pairs, and on a hit above a similarity threshold, return the cached answer in <100ms instead of running retrieval + a multi-second LLM call. Reported impact: up to 68.8% reduction in API calls and ~65× faster responses, with 60–85% hit rates in high-repetition workloads. Threshold tuning is the knob: 0.90–0.95 for high precision (you'd rather miss a cache hit than serve a subtly-wrong reused answer), 0.85–0.90 to maximize savings. Use a two-layer design — an exact key-value layer for literally-repeated queries, plus the vector layer for paraphrases. The trap: a too-loose threshold serves the answer to question A for question B; cache invalidation must also fire when underlying documents change.
Freshness is a policy, not a default. Three approaches, chosen by use case:
The subtle, high-signal point: deletion is the hard part. Batch pipelines that only upsert leave orphaned vectors — a deleted document's chunks keep getting retrieved and cited, which is both a correctness bug and a compliance risk. You need explicit deletion tracking (tombstones, version columns) and a reconciliation job. And you need version tracking: a 2021 policy chunk that contradicts the 2026 version must be down-ranked or filtered by updated_at, or the LLM will confidently cite the obsolete one.
Access control must be enforced inside retrieval, not as a post-filter on the LLM's answer. Stamp every chunk with tenant_id and acl_groups at ingest, and push those as pre-filters into the ANN query so the candidate set only ever contains chunks the requesting user may see. Post-filtering after generation is a leak waiting to happen — the model may already have quoted a forbidden chunk. For hourly-changing permissions, either evaluate ACLs at query time against a fast authz service (the index stores group IDs; the live check resolves user→group) or accept a bounded staleness window and reconcile. A leaked chunk is a compliance incident, so the design bias is: filter early, filter in the index, and fail closed.
Evaluation is what separates a demo from a system. Build a golden set — 30–50+ expert-annotated query→relevant-chunk (and query→answer) pairs per use case or failure mode, with <20% annotator disagreement on clear cases — and measure recall@k and precision-by-document-type for retrieval, and faithfulness/citation-correctness for generation (LLM-as-judge, ideally G-Eval chain-of-thought for better human correlation). Run this as a regression gate in CI on every change to chunking, embedding model, or prompt. In production, sample 10–20% of traces for detailed LLM-judge scoring and log basic metrics (tokens, cost, latency, cache-hit, hallucination flags) on 100%. The metric that catches the most real bugs is precision segmented by document type — aggregate recall can look great while one document class is silently broken.
A production-shaped online retrieval path: parallel BM25 + vector, RRF fusion, cross-encoder rerank, with ACL pre-filtering. This is the load-bearing core; ingestion is omitted for space but mirrors it (chunk → embed → upsert with metadata).
import asyncio
from dataclasses import dataclass
@dataclass
class Candidate:
chunk_id: str
text: str
rank: int # 1-based rank within its source list
async def vector_search(qvec, tenant_id, acl_groups, k=20) -> list[Candidate]:
# ACL + freshness pushed INTO the ANN query, not applied after.
rows = await qdrant.search(
vector=qvec, limit=k,
query_filter={"must": [
{"key": "tenant_id", "match": {"value": tenant_id}},
{"key": "acl_groups", "match": {"any": acl_groups}},
]},
)
return [Candidate(r.id, r.payload["text"], i + 1) for i, r in enumerate(rows)]
async def bm25_search(query, tenant_id, acl_groups, k=20) -> list[Candidate]:
rows = await opensearch.search(query, tenant_id, acl_groups, size=k)
return [Candidate(r["_id"], r["text"], i + 1) for i, r in enumerate(rows)]
def rrf_fuse(lists: list[list[Candidate]], k: int = 60) -> list[Candidate]:
scores, texts = {}, {}
for lst in lists:
for c in lst:
scores[c.chunk_id] = scores.get(c.chunk_id, 0.0) + 1.0 / (k + c.rank)
texts[c.chunk_id] = c.text
ranked = sorted(scores, key=scores.get, reverse=True)
return [Candidate(cid, texts[cid], i + 1) for i, cid in enumerate(ranked)]
async def retrieve(query, qvec, tenant_id, acl_groups, top_final=8) -> list[Candidate]:
# 1) Parallel keyword + semantic retrieval (don't serialize these).
bm25, vec = await asyncio.gather(
bm25_search(query, tenant_id, acl_groups, k=20),
vector_search(qvec, tenant_id, acl_groups, k=20),
)
# 2) Fuse — score-agnostic, no normalization needed.
fused = rrf_fuse([bm25, vec])
# 3) Rerank only the fused top-30 (cross-encoder is the latency cost center).
pairs = [(query, c.text) for c in fused[:30]]
rel = await reranker.score(pairs) # one batched cross-encoder call
reranked = [c for _, c in sorted(zip(rel, fused[:30]),
key=lambda x: x[0], reverse=True)]
return reranked[:top_final]Why it's shaped this way: BM25 and vector search are issued with asyncio.gather so their latencies overlap rather than add. ACL and tenant filters live in both queries, so a forbidden chunk never enters the candidate pool. RRF needs no score calibration between the two engines. The reranker — the expensive stage — sees only the fused top-30 in a single batched call, bounding its cost. The caller would wrap this with a semantic-cache check before retrieve and pass the final 8 chunks (with citations) into the LLM prompt.
| Component | Cost | Latency | Quality impact | Failure mode | What changes at scale |
|---|---|---|---|---|---|
| Chunking (256–512 tok) | Embedding spend ∝ chunk count | none (offline) | Wrong size → mid-concept splits, more hallucination | 128-tok fragments strand context | Re-chunk = full re-embed of 90M chunks |
| Embedding model | $/M tokens at ingest | ~5–20ms/query at serve | MTEB is a prior, not truth; in-domain wins | Wrong model = low recall everywhere | 90M-chunk re-embed is a multi-day batch job |
| Vector store | $/vector/month (RAM vs object storage) | ANN p95: ms in-RAM, higher on object storage | Filter-after-ANN drops recall | Single-node RAM ceiling (~600 GB) | pgvector → Qdrant/Turbopuffer at 10M+ |
| Hybrid + rerank | Reranker GPU/API calls | +200–400ms | +1–9% recall; catches exact matches | Reranking 200 candidates blows latency | Cap candidate set; batch rerank |
| Semantic cache | Cheap; saves up to 68.8% API calls | <100ms on hit | Loose threshold serves wrong answer | Stale cache after doc change | Invalidate on doc update; tune 0.85–0.95 |
| Freshness (CDC) | High operational overhead | sub-minute | Stale/obsolete citations | Orphaned vectors on delete | CDC the hot subset, batch the rest |
In prose: the cost model at 90M chunks is dominated by two lines — index residency (RAM is the expensive part; object-storage stores like Turbopuffer or int8 quantization cut it ~4×) and LLM generation. Target TTFT p90 < 2s and retrieval p95 in the low hundreds of ms; semantic-cache hits should land <100ms. The agentic multiplier is the cost trap to flag: LLM API prices fell ~80% from 2025–2026 (≈$30/M → ≈$0.40/M for GPT-4-level), but an agent making 50–200 calls per task turns a cheap per-token price into an expensive per-task bill — so per-task cost, not per-token, is the metric to optimize, which is exactly why model routing (cheap model for extraction/classification, expensive only for hard reasoning — e.g. Haiku over Sonnet for classification is ~12× cheaper at minimal quality loss) belongs in the design.
tenant_id, acl_groups, version, updated_at at ingest and push ACL/tenant as pre-filters into the ANN query so forbidden chunks never enter the candidate set — post-filtering after generation can leak quoted content. Permissions resolve at query time (index stores group IDs, a fast authz service resolves user→group), bounding staleness to seconds. Freshness is hybrid: CDC the hot subset, batch the rest. The thing that breaks first is deletion — upsert-only pipelines leave orphaned vectors that keep getting cited, so I need tombstones plus a reconciliation job, and I fail closed on any ACL-resolution error.Flashcard. 10M docs ≈ 90M chunks ≈ 600GB float32 HNSW index (or ~150GB int8). Below 10M chunks: pgvector. Above: Qdrant or Turbopuffer with ACL/freshness filters pushed into the ANN query.
Next: /rag for the retrieval internals, then /inference for the serving-cost optimizations that this design depends on.