Make a multi-step LLM pipeline that can be killed mid-flight, restarted hours later, and resume exactly where it left off — without double-charging a card or re-calling a $3 model.
ai-eng-wiki/examples/workflows/orchestrator.pyA durable workflow is a function whose execution state outlives the process running it. You write what looks like ordinary sequential code — call a model, charge a card, wait three days for a human to approve — and the runtime persists every step to a log so that a crash, deploy, or week-long pause loses nothing. The trick is a hard split: the workflow is deterministic glue (decisions, control flow) that can be re-run for free, and activities are the side-effecting calls (LLM, payment, email) whose results are recorded once and never re-executed on replay. On recovery the runtime replays the workflow from the top, but every activity that already finished returns its cached result from the event log instead of firing again. That is how you get "charge exactly once" out of a system that might run your code a hundred times. Everything else in this lesson — idempotency keys, retry policies, human-in-the-loop pauses — is bookkeeping around that one idea.
datetime.now() or a raw random() inside workflow code is a landmine. That single distinction separates "used Temporal once" from "understands it."The words first.
Step by step.
Remember this: the workflow is replayable glue; activities run once and are remembered.
Consider a three-step pipeline: (1) an LLM call that drafts a contract summary, (2) a payment capture, (3) a confirmation email. Each costs something — the LLM call burns tokens (say a $0.04 Opus call), the payment moves real money, the email annoys a real human. Now ask the interview question: the process dies after step 2 but before step 3. A naive "just retry the whole thing" re-runs the LLM (wasted $0.04, fine) and re-captures the payment (double charge, not fine).
The default delivery guarantee in any distributed system is at-least-once: when a call times out, you genuinely don't know if it succeeded, so the safe move is to retry — which means some calls happen twice. Exactly-once delivery is provably impossible over an unreliable network. What you can build is effectively-exactly-once execution: the call may be delivered twice, but the effect lands once, because the receiver deduplicates. That dedup is the whole game.
Idempotency is the property that re-applying an operation is a no-op. "Set balance to $100" is idempotent; "subtract $10" is not. For operations that aren't naturally idempotent (a charge, an email, an INSERT), you make them idempotent by attaching a stable idempotency key — a deterministic ID derived from the business event, not from the clock or a fresh UUID.
The receiver (Stripe, your DB, your email provider) keeps a table of seen keys. First request with key order-42-capture: do the work, record the key. Second request with the same key: return the stored result, do nothing. Stripe literally implements this; most payment and messaging APIs do. The art is choosing a key that is stable across retries (f"capture-{order_id}", not uuid4()) but unique across distinct events (don't reuse capture-{order_id} for a refund).
Symbols in plain words: key = a stable string naming this exact business action. seen = the provider's table mapping keys to results. effect = the real-world side effect (money moves).
Walk it: order #42, total $10.00. We build key = "capture-42".
seen → key absent → charges $10.00, writes seen["capture-42"] = {status: ok, amount: 1000}. Process crashes before we record success locally.key = "capture-42".seen → key present → returns {status: ok, amount: 1000}, charges nothing.Net money moved: $10.00, once. Without the key, attempt 2 charges again → $20.00. The key turned an at-least-once delivery into an exactly-once effect.
The subtlety interviewers love: the idempotency key has to be derivable on replay. If you generate it with uuid4() at call time, the replay generates a different key and the dedup fails. The key must be a pure function of inputs the workflow already has — which is exactly why workflow code must be deterministic (next section).
Temporal's mechanism is the clearest mental model, so learn it even if you never use Temporal. Every workflow execution is an append-only event history: WorkflowStarted, ActivityScheduled(draft_summary), ActivityCompleted(draft_summary, result="..."), TimerFired, SignalReceived(approval), and so on. The history is the durable state — there's no separate "save state" call.
On recovery, a worker performs deterministic replay: it runs your workflow function from the very first line. When the code reaches an activity call, the worker checks the history. If ActivityCompleted for that call is already in the log, it returns the cached result immediately without calling the activity — no LLM hit, no charge, no API call. Only when replay reaches a point past the recorded history does it actually schedule new work. This is why a workflow can sleep for three days or wait a year for a human and resume perfectly: nothing is held in process memory; everything is reconstructed from the log.
The non-negotiable contract: workflow code must be deterministic. Same history → same sequence of decisions, every replay. That bans, inside workflow code:
datetime.now(), time.time(), random(), uuid4() — these return different values each replay, diverging the path. Use the runtime's deterministic versions (workflow.now(), workflow.uuid4()) which are recorded in history.set or dict in a way whose order can change, or depending on map ordering across versions.Determinism is what makes replay free and correct. Break it and replay diverges — the worker tries to take a branch the history never recorded, and Temporal throws a non-determinism error. (Versioning APIs exist to evolve workflow code without breaking in-flight executions, but that's an advanced topic.)
Failures are normal; the question is what retries. You retry activities, not workflows. Re-running a non-deterministic workflow on every failure would re-execute side effects and diverge replay. Instead, the runtime retries the activity per a retry policy, and the workflow simply waits at that checkpoint until the activity finally succeeds (or exhausts retries).
A typical policy (Temporal defaults): initial interval 1s, backoff coefficient 2.0, maximum interval 100s, with optional maximum attempts and a list of non-retryable error types. So delays go 1s, 2s, 4s, 8s … capped at 100s. Exponential backoff prevents a flapping dependency from being hammered; jitter (randomizing each delay) prevents a thundering herd of synchronized retries after an outage. Mark errors like InvalidArgument or 400 BadRequest as non-retryable — retrying a malformed request just wastes money and latency. For LLM activities specifically: retry 429 and 5xx, do not retry a 400 from a bad prompt schema.
This is also where cost discipline lives. An agent that explores can spike token usage 4–15x; a durable workflow has a bounded, pre-computed call graph, so you can budget it: route cheap steps to a small model (Haiku-class) and reserve the expensive model for the steps that need it. Durability doesn't reduce per-call cost, but it removes the waste of re-doing completed work after a crash.
The feature that justifies durable execution to a skeptic is the multi-day pause. A contract workflow drafts a summary, then must wait for a human lawyer to approve before charging. With a normal process you'd need a database row, a polling job, and glue to rehydrate state when approval arrives — a pile of bespoke plumbing. With durable execution you write:
summary = await draft_summary(doc) # activity, recorded
approval = await wait_for_signal("approve") # workflow blocks here, durably
if approval.ok:
await capture_payment(order_id) # activity, idempotentThe wait_for_signal blocks the workflow with zero process memory held — the execution is suspended in the durable log. The worker can be redeployed, the machine recycled; when the approval signal arrives next Tuesday, a worker replays history, lands back exactly at the wait, and continues. LangGraph models the same thing as an interrupt that suspends a checkpointed graph until a human resumes it. This is genuinely hard to build correctly by hand and nearly free with a durable runtime — which is why it dominates real-world adoption (approvals, escalations, "wait for the user's reply").
Anthropic frames the design space as a spectrum: workflows orchestrate LLM calls through predefined code paths; agents let the model dynamically direct its own process at runtime. The litmus test: can you draw the flowchart before execution? Yes → workflow. No → agent. Durable execution is most natural for the workflow end, where the call graph is known and replay is clean. For agents (covered in /agents), the control flow itself is model-decided, which complicates replay — we tackle that in section 6. The pragmatic production shape, per both Anthropic and Redis's analysis, is hybrid: a deterministic, durable supervisor that routes to bounded autonomous specialists.
Below is a self-contained durable orchestrator — the shape of examples/workflows/orchestrator.py. It implements the core ideas without a heavyweight runtime: an append-only event log, replay that short-circuits completed steps, idempotent side effects, and retry with backoff + jitter. Run it, kill it mid-way (Ctrl-C after the draft), and run it again — it resumes without re-calling the LLM or re-charging.
import json, os, time, random, hashlib
from pathlib import Path
LOG = Path("workflow_events.jsonl")
# ---- durable event log: append-only, replayable -------------------------
def append_event(event: dict) -> None:
with LOG.open("a") as f:
f.write(json.dumps(event) + "\n")
f.flush(); os.fsync(f.fileno()) # durability: hit disk before returning
def load_history() -> dict:
"""Map step_id -> recorded result. This is the replay cache."""
done = {}
if LOG.exists():
for line in LOG.read_text().splitlines():
ev = json.loads(line)
if ev["type"] == "ActivityCompleted":
done[ev["step_id"]] = ev["result"]
return done
# ---- activity runner: retry + backoff, write-once result ----------------
def run_activity(step_id: str, fn, history: dict, *,
max_attempts=5, base=1.0, cap=30.0):
if step_id in history: # REPLAY: cached, do not re-run
print(f"[replay] {step_id} -> cached")
return history[step_id]
for attempt in range(1, max_attempts + 1):
try:
result = fn() # the only place side effects happen
append_event({"type": "ActivityCompleted",
"step_id": step_id, "result": result})
history[step_id] = result
return result
except RetryableError as e:
if attempt == max_attempts:
raise
delay = min(cap, base * 2 ** (attempt - 1))
delay += random.uniform(0, delay * 0.25) # jitter
print(f"[retry] {step_id} attempt {attempt} failed ({e}); "
f"sleeping {delay:.1f}s")
time.sleep(delay)
class RetryableError(Exception): ...
# ---- side effects (idempotent) -----------------------------------------
_charged = {} # stands in for the payment provider's "seen keys" table
def idem_key(*parts) -> str: # STABLE across replays
return hashlib.sha256("|".join(map(str, parts)).encode()).hexdigest()[:16]
def draft_summary(doc_id: str) -> dict:
if random.random() < 0.3: # simulate a flaky LLM 5xx
raise RetryableError("LLM 503")
print(f" >> calling LLM for doc {doc_id} ($0.04)")
return {"summary": f"summary-of-{doc_id}"}
def capture_payment(order_id: str, cents: int) -> dict:
key = idem_key("capture", order_id) # derived from inputs, not uuid4
if key in _charged: # provider-side dedup
print(f" >> capture {key} already seen, no double charge")
return _charged[key]
print(f" >> charging {cents}c for order {order_id}")
_charged[key] = {"status": "ok", "cents": cents, "key": key}
return _charged[key]
# ---- the workflow: deterministic glue over activities -------------------
def run_workflow(doc_id: str, order_id: str):
history = load_history() # rebuild state from the log
summary = run_activity(f"draft:{doc_id}",
lambda: draft_summary(doc_id), history)
charge = run_activity(f"charge:{order_id}",
lambda: capture_payment(order_id, 1000), history)
print(f"[done] {summary['summary']} | charged {charge['cents']}c once")
if __name__ == "__main__":
run_workflow("doc-7", "order-42")What each piece buys you. The fsync after every append is the durability boundary — without it, a crash can lose the last record and you'd re-charge. load_history + the step_id in history short-circuit is replay: completed steps return cached results, never re-calling the LLM or the payment API. idem_key derives from inputs, so the same key is regenerated on replay and the provider's dedup table (_charged) catches the duplicate. run_activity owns retries with exponential backoff and jitter; the workflow function stays pure glue — note it contains no now(), no random(), no I/O directly. This is a teaching model; in production you'd reach for Temporal or a LangGraph checkpointer (section 5) rather than hand-rolling the log, but the moving parts are identical.
| Approach | Durability model | Best for | Latency / cost overhead | Sharpest failure mode |
|---|---|---|---|---|
| Temporal | Event-sourced replay; activities at-least-once | Long-running, stateful, code-first workflows; multi-day human-in-loop | ~ms per event persisted; infra to operate a cluster | Non-determinism in workflow code → replay divergence |
| LangGraph checkpointer | Snapshot state per node (Postgres/Redis/SQLite) | Agentic + workflow graphs, cycles, interrupts | Checkpoint write per node | In-memory default loses everything on restart; large state blobs |
| Airflow (DAG) | Task state in metadata DB; stateless tasks | Batch ETL, scheduled pipelines | Scheduler overhead; coarse-grained | Context must go through external storage; weak for sub-second/LLM loops |
| Dagster (assets) | Asset materialization + lineage | Data products with governance/lineage | Similar to Airflow | Asset-centric model awkward for free-form orchestration |
| Hand-rolled (queue + log) | Whatever you build | Tiny scope, full control, no new infra | Lowest if simple; explodes with edge cases | You will reinvent idempotency/replay and get an edge case wrong |
Cost and latency. Durability adds a persistence write per step (sub-millisecond to a few ms) — negligible next to an LLM call (hundreds of ms to seconds). The real cost is operational: running a Temporal cluster, or sizing a Postgres/Redis checkpoint store. The savings are in avoided rework: a 10-step pipeline that crashes at step 9 doesn't redo 8 LLM calls. And recall the reliability math — a 10-step process at 99% per-step success compounds to ~90% end-to-end; durable retries claw most of that back by surviving the transient 1%.
Quality and what changes at scale. At low volume, a hand-rolled log is fine. At 1,000s of runs/day you need: durable state stores (Redis/Postgres, never in-memory defaults), backpressure on activity queues, retry budgets so a flapping dependency doesn't melt your bill, and observability — OpenTelemetry spans per LLM call / tool / state transition, structured queryable logs, and annotation queues routing traces to domain experts for correctness (89% of orgs have agent observability, but only ~52% close the evaluation loop). See /system-design for the storage tiers.
Failure modes to name in an interview. (1) Idempotency key generated non-deterministically → replay creates a new key → double charge. (2) Side effect inside workflow code instead of an activity → re-executes on replay. (3) Retrying non-retryable errors (a 400) → wasted spend, never succeeds. (4) Unbounded retries with no jitter → thundering herd after an outage. (5) In-memory checkpointer in "prod" → silent total state loss on the first restart.
capture-{order_id} so the provider dedups a retry, and make the email idempotent the same way. Net: charge once, email once, regardless of where the crash lands.ActivityCompleted event is already in the history, the worker returns the logged result directly and skips the actual call. Only work past the recorded history is newly scheduled. The rule: workflow code must be deterministic — same history, same path, every replay. No now(), random(), uuid4(), env reads, or direct I/O in workflow code; those go in activities (recorded) or use the runtime's deterministic equivalents. Break determinism and replay diverges into a branch the log never recorded.uuid4(). Generated fresh each replay → dedup never matches → double charge. Derive the key from stable inputs (order_id), always.fsync. "Persisted" but still in the OS page cache = lost on power failure. Durability means it hit stable storage before you returned success.400/InvalidArgument will never succeed; retrying it 5x burns money and latency. Classify errors; only retry 429/5xx/timeouts.Flashcard. Durable execution = deterministic workflow (replayable for free) + side-effecting activities (run once, result logged). Replay skips completed activities by reading the event history; idempotency keys turn at-least-once delivery into exactly-once effects.
Next: /workflows/observability — instrumenting durable workflows with OpenTelemetry spans and closing the evaluation loop with annotation queues.