Senior to Staff · architecture patterns · 30 min

Architecture patterns
for AI systems.

Five patterns cover almost everything teams are building right now: chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer. Tonight: what each one costs, which failure mode each one buys you, and the honest answer that most systems sold as agents should have been one of these.

Ehsan Gazar
Principal Engineer · 16 years in production
Where we're going · 30 minutes

The run sheet.

0–3The reframeworkflow or agent, who drives
3–5The autonomy laddersix rungs, left to right
5–17The five patternsshape, cost, failure mode
17–21Gates and compositionwhat goes between the steps
21–26Graduating to an agentthree gates, and the caps
26–30Cost, antipatterns, nextthe Staff lens
By the end tonight

You'll be able to…

1Name the pattern a task actually wants, instead of reaching for an agent because that's the word in the ticket.
2Price each pattern in calls, tokens, and p95 latency, and say out loud which failure mode it introduces.
3Defend the choice at Staff level: gates between steps, caps on every loop, and the three gates a task must clear before it earns agency.
The reframe

Most things called an agent
should have been a workflow.

The word "agent" got attached to anything with a model in it, and teams bought non-determinism they never needed. The real question is not how smart the model is. It's who decides what happens next: your code, or the model.

💡 In plain English

A workflow is a recipe you wrote. An agent is a cook you hired. Both can produce dinner. Only one of them can decide, at 2am, to reorganise your kitchen. Hire the cook when the recipe genuinely can't be written, not before.

The one distinction that matters

Who drives
control flow?

In a workflow, your code owns the path and the model fills in steps. In an agent, the model owns the path and your code executes what it asks for.

Workflow

Path is fixed at write time. Testable per node, bounded by construction, cost is knowable before you ship.

Agent

Path is decided at run time. Every axis is unbounded until you bound it, and the worst case is "it never stops".

Workflow · your code owns the path
flowchart LR
  wIn["Input"]:::in --> wS1["Step 1"]:::step
  wS1 --> wG{"Check"}:::q
  wG --> wS2["Step 2"]:::step
  wS2 --> wOut["Output"]:::out
  classDef in fill:#DCEBFE,stroke:#0D1B33,color:#0D1B33;
  classDef step fill:#ffffff,stroke:#0D1B33,color:#0D1B33;
  classDef q fill:#FEF3C7,stroke:#0D1B33,color:#0D1B33;
  classDef out fill:#D6F5E3,stroke:#0D1B33,color:#0D1B33;
Agent · the model owns the path
flowchart LR
  aIn["Goal"]:::in --> aM["Model decides"]:::model
  aM -->|act| aT["Run a tool"]:::step
  aT -->|observe| aM
  aM -->|done| aOut["Output"]:::out
  classDef in fill:#DCEBFE,stroke:#0D1B33,color:#0D1B33;
  classDef step fill:#ffffff,stroke:#0D1B33,color:#0D1B33;
  classDef model fill:#EDE9FE,stroke:#0D1B33,color:#0D1B33;
  classDef out fill:#D6F5E3,stroke:#0D1B33,color:#0D1B33;
The map · six rungs

The autonomy ladder.

rung 0
Single call
one prompt, one answer
rung 1
Chain
ordered steps, gated
rung 2
Router
classify, then dispatch
rung 3
Parallel
fan out, aggregate
rung 4
Orchestrator
plan decided at run time
rung 5
Agent
model owns the loop
Every rung to the right buys capability with tokens, latency, and blast radius. Reach left. Buy agency only when a gate forces it.
Pattern 1 of 5 · chaining

Prompt chaining:
split, then gate.

Break the task into ordered steps and put a check between them. Validate the outline before you spend tokens writing the essay.

Use when

The task has clean, checkable sub-steps and each step's output is the next step's input.

Breaks when

The steps are actually independent, parallelise instead, or the shape isn't known up front, orchestrate instead.

cost: N sequential calls · latency adds up · keep it short
flowchart LR
  chIn["Brief"]:::in --> chA["Draft outline"]:::model
  chA --> chG{"Valid?"}:::q
  chG -->|no| chX["Reject early"]:::bad
  chG -->|yes| chB["Write from outline"]:::model
  chB --> chC["Tighten and proof"]:::model
  chC --> chOut["Final"]:::out
  classDef in fill:#DCEBFE,stroke:#0D1B33,color:#0D1B33;
  classDef q fill:#FEF3C7,stroke:#0D1B33,color:#0D1B33;
  classDef model fill:#ffffff,stroke:#0D1B33,color:#0D1B33;
  classDef bad fill:#FEE4E2,stroke:#0D1B33,color:#0D1B33;
  classDef out fill:#D6F5E3,stroke:#0D1B33,color:#0D1B33;
Pattern 2 of 5 · routing

Routing:
classify, then dispatch.

A small, fast model reads the request first and sends it down the right-sized path. Cheap model for the easy ones, frontier model for the hard ones, a dedicated flow for refunds.

Use when

Inputs fall into distinct kinds that each want different handling or a different model tier. The easiest cost win you get.

Breaks when

The classifier is wrong. A misroute sends a hard case down the cheap path and nobody notices. Measure routing accuracy.

cost: 1 small call plus the chosen flow · usually a net saving
flowchart LR
  rtIn["Request"]:::in --> rtR{"Classify"}:::q
  rtR -->|faq| rtS["Small model"]:::cheap
  rtR -->|refund| rtF["Dedicated flow"]:::step
  rtR -->|hard| rtL["Frontier model"]:::model
  rtS --> rtOut["Response"]:::out
  rtF --> rtOut
  rtL --> rtOut
  classDef in fill:#DCEBFE,stroke:#0D1B33,color:#0D1B33;
  classDef q fill:#FEF3C7,stroke:#0D1B33,color:#0D1B33;
  classDef cheap fill:#D6F5E3,stroke:#0D1B33,color:#0D1B33;
  classDef step fill:#ffffff,stroke:#0D1B33,color:#0D1B33;
  classDef model fill:#EDE9FE,stroke:#0D1B33,color:#0D1B33;
  classDef out fill:#FEE4E2,stroke:#0D1B33,color:#0D1B33;
Pattern 3 of 5 · parallelization

Parallelization:
fan out, aggregate.

Two flavours. Sectioning splits one job into independent parts. Voting runs the same job several times and takes the consensus. Faster, and often better.

Use when

Subtasks don't depend on each other, or independent votes raise reliability on a hard judgement call.

Breaks when

Branches secretly depend on one another, or the merge is naive and quietly averages away real disagreement.

cost: N× tokens, about 1× latency, the calls run concurrently
flowchart LR
  paIn["Document"]:::in --> pa1["Summarise"]:::model
  paIn --> pa2["Extract risks"]:::model
  paIn --> pa3["List next actions"]:::model
  pa1 --> paM["Merge or vote"]:::win
  pa2 --> paM
  pa3 --> paM
  paM --> paOut["Result"]:::out
  classDef in fill:#DCEBFE,stroke:#0D1B33,color:#0D1B33;
  classDef model fill:#ffffff,stroke:#0D1B33,color:#0D1B33;
  classDef win fill:#FEF3C7,stroke:#0D1B33,color:#0D1B33;
  classDef out fill:#D6F5E3,stroke:#0D1B33,color:#0D1B33;
Pattern 4 of 5 · orchestrator-workers

Orchestrator-workers:
the plan is run-time.

A planner splits work it cannot know in advance, spawns a worker per subtask, then synthesizes. The branches are decided while it runs. This is the first real taste of agency.

Use when

The number and shape of subtasks depend on the input: multi-file edits, research over an unknown set of sources.

Breaks when

The plan runs away with no cap on subtasks, or the synthesis step drowns in worker output and loses the thread.

cost: 1 planner + N workers + 1 synthesis · bound N
flowchart LR
  orIn["Task"]:::in --> orP["Planner"]:::model
  orP --> orW1["Worker 1"]:::step
  orP --> orW2["Worker 2"]:::step
  orP --> orW3["Worker n, capped"]:::step
  orW1 --> orS["Synthesize"]:::win
  orW2 --> orS
  orW3 --> orS
  orS --> orOut["Result"]:::out
  classDef in fill:#DCEBFE,stroke:#0D1B33,color:#0D1B33;
  classDef model fill:#EDE9FE,stroke:#0D1B33,color:#0D1B33;
  classDef step fill:#ffffff,stroke:#0D1B33,color:#0D1B33;
  classDef win fill:#FEF3C7,stroke:#0D1B33,color:#0D1B33;
  classDef out fill:#D6F5E3,stroke:#0D1B33,color:#0D1B33;
Pattern 5 of 5 · evaluator-optimizer

Evaluator-optimizer:
critique and revise.

Generate, score against a rubric, revise, loop until it passes. Quality that improves without a human in every turn. Always cap the loop, or a $5 task quietly becomes a $50 one.

Use when

There's a clear rubric and revision reliably improves the draft: translation, code that must pass tests, structured extraction.

Breaks when

The evaluator can't tell better from worse, so it thrashes, or the loop has no cap and burns budget on cosmetic edits.

cost: 2 calls per round × rounds · the cap is the budget
flowchart LR
  evIn["Task"]:::in --> evG["Generate draft"]:::model
  evG --> evE{"Score vs rubric"}:::q
  evE -->|fail, under cap| evR["Revise"]:::model
  evR --> evE
  evE -->|pass or cap hit| evOut["Ship"]:::out
  classDef in fill:#DCEBFE,stroke:#0D1B33,color:#0D1B33;
  classDef model fill:#ffffff,stroke:#0D1B33,color:#0D1B33;
  classDef q fill:#FEF3C7,stroke:#0D1B33,color:#0D1B33;
  classDef out fill:#D6F5E3,stroke:#0D1B33,color:#0D1B33;
The two lines that keep the bill finite

Every fan-out and every loop
carries its cap in the code.

# orchestrator: the planner decides the branches, you decide the ceiling
plan = orchestrator(task)              # -> list of subtasks
assert len(plan) <= MAX_SUBTASKS       # 1. bound the fan-out
results = gather(*[worker(s) for s in plan])
answer  = synthesise(task, results)

# evaluator-optimizer: generate, critique, revise, ALWAYS capped
draft = generate(task)
for _ in range(MAX_ROUNDS):             # 2. bound the loop
    v = evaluate(draft, rubric)
    if v.passed: break
    draft = revise(draft, v.feedback)
return draft
If you can't point at the constant, you don't have a budget. You have a leak.
What goes between the steps

A pattern is only
as good as its gates.

~free
Deterministic check
Schema, type, regex, range, required fields, does it parse. No model call. Catches malformed output instantly.
1 call
LLM as judge
A model scores the output against a rubric: faithfulness, tone, completeness. For quality you can't express as a rule.
↻
Ground-truth compare
Check the answer against a source document or a golden set. The main defence against confidently wrong output.
👤
Human in the loop
A person approves before the action commits. Reserve it for irreversible or high-stakes steps, it costs latency and attention.
Escalate left to right, and only when the cheaper gate genuinely can't catch the failure.
Real systems, honestly

Patterns compose. Production
is a tree, not one clever loop.

flowchart TB
  cIn["Request"]:::in --> cR{"Router"}:::q
  cR -->|simple| cS["Single call"]:::step
  cR -->|structured| cC["Chain with gates"]:::step
  cR -->|research| cO["Orchestrator"]:::model
  cO --> cP["Parallel workers"]:::step
  cS --> cE{"Evaluator"}:::q
  cC --> cE
  cP --> cE
  cE -->|fail| cC
  cE -->|pass| cOut["Response"]:::out
  classDef in fill:#DCEBFE,stroke:#0D1B33,color:#0D1B33;
  classDef q fill:#FEF3C7,stroke:#0D1B33,color:#0D1B33;
  classDef step fill:#ffffff,stroke:#0D1B33,color:#0D1B33;
  classDef model fill:#EDE9FE,stroke:#0D1B33,color:#0D1B33;
  classDef out fill:#D6F5E3,stroke:#0D1B33,color:#0D1B33;
Build the smallest composition that clears the task, and keep each node testable on its own. Complexity you can't test in pieces is complexity you can't debug in production.
Graduating to an agent

Three gates.
All three, or stay put.

Most features stop at the first gate, and that is a good outcome, not a failure of ambition.

Gate 1 · unknowable steps

Can you write the steps in code ahead of time? If yes, a workflow wins.

Gate 2 · needs a loop

Must it act, observe the result, and decide the next move on the fly? If a fixed chain would do, you don't need an agent.

Gate 3 · worth the risk

Is the value worth the tokens, latency, and blast radius, and can you afford the guardrails and evals it demands?

flowchart TB
  gStart["A task to build"]:::in --> g1{"Steps known in advance?"}:::q
  g1 -->|yes| gWF["Use a workflow"]:::wf
  g1 -->|"no, open-ended"| g2{"Must it act, observe, decide?"}:::q
  g2 -->|no| gWF
  g2 -->|yes| g3{"Worth cost, latency, risk?"}:::q
  g3 -->|"not yet"| gWF
  g3 -->|yes| gAG["Agent, with guardrails and evals"]:::ag
  classDef in fill:#DCEBFE,stroke:#0D1B33,color:#0D1B33;
  classDef q fill:#FEF3C7,stroke:#0D1B33,color:#0D1B33;
  classDef wf fill:#D6F5E3,stroke:#0D1B33,color:#0D1B33;
  classDef ag fill:#EDE9FE,stroke:#0D1B33,color:#0D1B33;
If you do ship an agent

Put a ceiling
on every axis it can run away on.

AxisGuardrail
IterationsHard max-steps, and break on no-progress.
BudgetToken and dollar ceiling per request. Kill the run on breach.
ToolsAllow-list, least privilege, no raw shell.
Irreversible actsHuman approval, or a dry-run first.
TimePer-tool and per-request timeouts.
Input and outputStructured tool schemas. Validate every argument before it runs.
FailureA typed fallback path. Never a raw stack trace to the user.
An unguarded agent isn't a bold architecture. It's an unbounded bill with a demo attached.
Think like a Staff engineer · napkin math

What each rung actually costs.

ShapeModel callsRel. tokensRel. p95 latencyBlast radius
Single call11×1×tiny
3-step chain3~3×~3×small, bounded
Router plus flow2~1–2×~1.5×small
Parallel, 4 branches4~4×~1× concurrentmedium cost
Agent, ~10 turns10–30+10–30×10×+large, unbounded
Order of magnitude, not benchmarks. Price the worst case: cost of one turn × your loop cap. That's the number that goes in the budget.
The six ways this goes wrong

Antipatterns you'll
recognise in a design review.

Monolithic agent
One giant loop with 30 tools doing everything. Impossible to test, reason about, or bound.
Over-engineered planning
Multi-agent choreography for a job a three-step chain would nail.
Premature agency
An agent where a router would do. You bought non-determinism you didn't need, and now you have to guard it.
Unbounded loop
Generate-critique or agent loop with no cap. A $5 task quietly bills $50 the day inputs get weird.
Missing observability
No traces, no evals. You learn it's broken from users, not dashboards.
No fallback path
One bad model call and the raw error reaches the user. No retry, no default, no graceful degrade.
The cure for all six: start simple, make it observable, cap every loop, add agency only when a gate demands it.
The slide to screenshot

Pick the pattern.

If the task is…Reach forBecause
Ordered steps, each checkableChainingGate between steps, fail early and cheap.
Different inputs, different handlingRoutingRight-size the model per request.
Independent subtasks, known up frontParallelizationLatency down, quality up via voting.
Subtasks unknowable until run timeOrchestrator-workersThe planner decides the branches.
Quality needs iteration to a barEvaluator-optimizerCritique and revise, capped.
Open-ended, must loop, worth the riskAgentOnly after all three gates clear.
Recap · one sentence

Agency is a cost you buy,
not a level you graduate to.

Five patterns cover almost everything: chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer. Reach left on the ladder. Gate between steps with the cheapest check that catches the failure. Cap every loop, price the worst case, and instrument it before you scale. Say those out loud in a review and you're designing, not guessing.

Go deeper

This was the map.
The cohort is where you build it.

Production-Ready Systems with LLMs and Agents: An Intensive for Engineers. Six weeks, twelve live sessions, graded projects where you ship real systems, not notebooks. Week 1 is exactly this: all five patterns, implemented, gated, and priced.

★
Take the cheat sheet with you
Every pattern on this deck, with runnable code and the gate table, at hub.gazar.dev.
workflow patterns cheat sheet
Enrol
maven.com/gazar/
production-ready-systems-with-llms-and-agents-an-intensive-for-engineers
link is in the chat
Ehsan Gazar
Principal Engineer · 500+ mentees · me@gazar.dev