Lightning Lesson · Tue 16 June 2026 · 30 min · live

System design for AI agents.
For engineers.

Every team is wiring an LLM into production, and most of the guidance is written for product managers. Tonight is the engineer's mental model: what's ordinary distributed-systems work, what genuinely breaks, and the few decisions that decide whether it survives real traffic.

Ehsan Gazar

Principal Engineer · 16 years in production · 465 registered

Where we're going · 30 minutes

The run sheet.

0–3The reframewhy engineers got skipped

3–9Common shapesRAG, agents, routing, multi-agent

9–13What's ordinaryyou already solve most of it

13–20What genuinely breaksa non-deterministic dependency

20–26The four decisionsboundaries, state, failure, eval

26–30Reliability, cost, what's nextStaff lens + the cohort

By the end of tonight

You'll be able to…

1Tell normal distributed-systems work from what is genuinely new, so you spot which parts you already solve and which break your assumptions.

2Name the decisions that matter with an LLM in the path: boundaries, state, failure modes, and evaluation.

3Reason about reliability and cost like a Staff engineer: non-determinism, retries, latency, and token cost as first-class constraints, not afterthoughts.

The reframe

Senior engineers are shipping
agent systems on instinct.

The hype is written for the people who buy, not the people who page. So strong engineers wire an LLM into a flow, it demos beautifully, and then it meets real traffic. An agent is not a new kind of magic. It's a service with an unusually weird dependency.

💡 In plain English

You already know how to build reliable systems around a slow, expensive, flaky dependency. An LLM is exactly that, plus one new twist: it can be confidently wrong. Most of tonight is naming which of your existing instincts still apply, and the few places they don't.

Common shapes · 1 of 4

RAG: give it
the facts.

Retrieval-Augmented Generation. Fetch the relevant text at query time and put it in the context, so the model answers from your data instead of its training.

Use it when

the answer lives in your docs or database, not the model's weights. The default first LLM system, and retrieval quality is the whole game.

flowchart LR
  rgQ["User question"]:::in --> rgEMB["Embed query"]:::step
  rgEMB --> rgVDB[("Vector store")]:::store
  rgVDB --> rgK["Top matches"]:::step
  rgQ --> rgP["Prompt plus context"]:::win
  rgK --> rgP
  rgP --> rgLLM["LLM"]:::model
  rgLLM --> rgA["Grounded answer"]:::out
  classDef in fill:#DCEBFE,stroke:#0D1B33,color:#0D1B33;
  classDef step fill:#ffffff,stroke:#0D1B33,color:#0D1B33;
  classDef store fill:#EDE9FE,stroke:#0D1B33,color:#0D1B33;
  classDef win fill:#FEF3C7,stroke:#0D1B33,color:#0D1B33;
  classDef model fill:#D6F5E3,stroke:#0D1B33,color:#0D1B33;
  classDef out fill:#FEE4E2,stroke:#0D1B33,color:#0D1B33;

Common shapes · 2 of 4

Tool-calling agent:
reason, act, repeat.

The model thinks, calls a tool, reads the result, and loops until the task is done. This is the ReAct pattern, and the core of most agents.

Use it when

the task needs live data or real actions, not just generated text. Always bound the loop with a step cap.

flowchart LR
  taU["Task"]:::in --> taM["LLM reasons"]:::model
  taM -->|needs data| taT["Call a tool"]:::step
  taT --> taO["Observation"]:::win
  taO --> taM
  taM -->|done| taANS["Answer"]:::out
  classDef in fill:#DCEBFE,stroke:#0D1B33,color:#0D1B33;
  classDef step fill:#ffffff,stroke:#0D1B33,color:#0D1B33;
  classDef win fill:#FEF3C7,stroke:#0D1B33,color:#0D1B33;
  classDef model fill:#D6F5E3,stroke:#0D1B33,color:#0D1B33;
  classDef out fill:#FEE4E2,stroke:#0D1B33,color:#0D1B33;

Common shapes · 3 of 4

Routing:
pick the right path.

A cheap classifier reads the request first, then dispatches it to the handler that fits: a small model, a large model, or a tool.

Use it when

inputs vary a lot in difficulty or kind. The easiest win on cost and latency without hurting quality.

flowchart LR
  rtIN["Request"]:::in --> rtR{"Router"}:::q
  rtR -->|simple| rtS["Small model"]:::model
  rtR -->|hard| rtL["Large model"]:::model
  rtR -->|lookup| rtK["RAG or tool"]:::step
  rtS --> rtOUT["Response"]:::out
  rtL --> rtOUT
  rtK --> rtOUT
  classDef in fill:#DCEBFE,stroke:#0D1B33,color:#0D1B33;
  classDef q fill:#FEF3C7,stroke:#0D1B33,color:#0D1B33;
  classDef step fill:#ffffff,stroke:#0D1B33,color:#0D1B33;
  classDef model fill:#D6F5E3,stroke:#0D1B33,color:#0D1B33;
  classDef out fill:#FEE4E2,stroke:#0D1B33,color:#0D1B33;

Common shapes · 4 of 4

Orchestrator
and workers.

A lead agent splits the job into subtasks, runs specialized workers (often in parallel), and synthesizes their output into one result.

Use it when

the task is too big or varied for a single prompt. The common multi-agent shape, but reach for it last, the coordination is the cost.

flowchart LR
  orO["Orchestrator"]:::model --> orA["Research"]:::step
  orO --> orB["Draft"]:::step
  orO --> orC["Review"]:::step
  orA --> orS["Synthesize"]:::win
  orB --> orS
  orC --> orS
  orS --> orOUT["Result"]:::out
  classDef step fill:#ffffff,stroke:#0D1B33,color:#0D1B33;
  classDef win fill:#FEF3C7,stroke:#0D1B33,color:#0D1B33;
  classDef model fill:#D6F5E3,stroke:#0D1B33,color:#0D1B33;
  classDef out fill:#FEE4E2,stroke:#0D1B33,color:#0D1B33;

The whole talk in one frame

Three buckets.

1

Ordinary

Timeouts, retries, idempotency, queues, rate limits, caching, observability. You have this.

most of the system

2

Genuinely breaks

A dependency that is non-deterministic and can be plausibly, silently wrong.

the new twist

3

The few decisions

Boundaries, state, failure modes, evaluation. Where design effort actually pays.

earn your title here

Bucket 1 · what's ordinary

An agent is a service that calls
a slow, expensive, flaky dependency.

Strip the word "AI" and look at the shape. A request comes in, you call a downstream that takes seconds and sometimes errors, you handle that, you return. Every tool you already reach for still applies.

Timeouts & retries

It's a network call to a flaky service. Bound it, retry it, back off.

Idempotency & queues

Don't double-charge on a retry. Push slow work off the request path.

Caching & observability

Cache stable answers. Log every call, its cost, and its latency.

Bucket 2 · what genuinely breaks

A normal service versus
an LLM in the path.

Property	Normal service	LLM in the path
Same input, same output	yes, deterministic	no, varies every call
How it signals failure	throws an error you catch	returns plausible text, no error
Cost driver	per request	per token, on every turn
Latency	milliseconds	seconds
Correctness	testable with assertions	probabilistic, must be evaluated

One row is the whole problem: it can be wrong with no exception attached.

Decision · state

State is the
context window.

The model has no memory between turns. Everything it knows for this request is what you assemble into the window: the system prompt, the history, what you retrieve, and the tools you expose.

💡 In plain English

The context window is the model's whole desk for this one task. What you put on the desk, and what you sweep off when it gets full, is a design decision, not a default.

flowchart LR
  stSP["System prompt"]:::in --> stCTX["Context window
all the model sees this turn"]:::win
  stHIS["Conversation history"]:::in --> stCTX
  stRET["Retrieved context"]:::in --> stCTX
  stTOOL["Tool definitions"]:::in --> stCTX
  stCTX --> stM["LLM"]:::model
  stM --> stOUT["Response or tool call"]:::out
  classDef in fill:#DCEBFE,stroke:#0D1B33,color:#0D1B33;
  classDef win fill:#FEF3C7,stroke:#0D1B33,color:#0D1B33;
  classDef model fill:#D6F5E3,stroke:#0D1B33,color:#0D1B33;
  classDef out fill:#FEE4E2,stroke:#0D1B33,color:#0D1B33;

Bucket 3 · the decisions that matter

Four calls decide
whether it works in production.

Boundaries

What the model is allowed to do versus what your code decides. Keep it out of the trust path; validate every tool call before it runs.

State

The context window is the model's entire memory for this request. What you assemble, retrieve, and truncate is a design decision, not a default.

Failure modes

Design for "wrong but confident": schema validation, guardrails, bounded loops, fallbacks, and a human in the loop where the blast radius is real.

Evaluation

You can't unit-test a probabilistic system the old way. Evals and golden sets, offline and online, so a prompt change doesn't silently regress.

Where the four decisions land

The agent loop, annotated.

state

Assemble context

system prompt, history, retrieved docs, tool defs

model

Model plans

proposes the next tool call

boundary

Validate & run

check the call, then execute it

eval

Update & check

score the step, then loop or stop

The arrow back from "check" to "context" is where your step cap lives. No cap means a runaway and an unbounded bill.

Decision · failure modes

Handling "wrong but confident."

flowchart LR
  trA["Model output"]:::out --> trB{"Can code verify it"}:::q
  trB -->|yes| trC["Validate vs schema or rules"]:::ok
  trC --> trE["Proceed"]:::ok
  trB -->|no| trD{"High blast radius"}:::q
  trD -->|yes| trF["Human in the loop"]:::warn
  trD -->|no| trG["Guardrail and fallback"]:::ok
  classDef q fill:#FEF3C7,stroke:#0D1B33,color:#0D1B33;
  classDef ok fill:#D6F5E3,stroke:#0D1B33,color:#0D1B33;
  classDef warn fill:#FEE4E2,stroke:#0D1B33,color:#0D1B33;
  classDef out fill:#DCEBFE,stroke:#0D1B33,color:#0D1B33;

Never let unverified, high-impact output act on its own. Verify in code, gate it, or put a human on it.

Think like a Staff engineer

Reliability and cost are
design inputs, not afterthoughts.

↻Non-determinism: pin what you can (low temperature, strict output schemas), validate everything you can't, and never trust a tool argument unread.

⛔Retries and loops: tool calls must be idempotent and the loop must be bounded. An agent with no step cap is an unbounded bill and a runaway.

⏱Latency: calls take seconds, not milliseconds. Stream, parallelize independent calls, and cache the stable parts of the context.

$Token cost: you pay per token, so longer context is slower and pricier on every turn. Budget it the way you budget p99 latency.

Recap · one sentence

An agent is a distributed system
with a confident, costly dependency
in the hot path.

Most of it is skills you already have. The leverage is in four decisions: boundaries, state, failure modes, evaluation, plus treating non-determinism, retries, latency, and token cost as first-class. Name those out loud and you're designing, not guessing.

Go deeper · the cohort

Tonight was the map.
The cohort is where you build it.

Production-Ready Systems with LLMs and Agents: An Intensive for Engineers. Six weeks, twelve live sessions (Tue/Thu), five graded projects where you ship real agent systems, not notebooks. Jul 13 to Aug 23.

★

Exclusive code: system-design-ai

20% off, so $1,200 instead of $1,500. The same code is in your recording email.

early-bird through Jul 1 · capped at 12

Enrol

maven.com/gazar/
production-ready-systems-with-llms-and-agents-an-intensive-for-engineers

link is in the chat

Ehsan Gazar

Principal Engineer · 500+ mentees · me@gazar.dev

System design for AI agents.For engineers.

The run sheet.

You'll be able to…

Senior engineers are shippingagent systems on instinct.

RAG: give itthe facts.

Tool-calling agent:reason, act, repeat.

Routing:pick the right path.

Orchestratorand workers.

Three buckets.

An agent is a service that callsa slow, expensive, flaky dependency.

A normal service versusan LLM in the path.

State is thecontext window.

Four calls decidewhether it works in production.

The agent loop, annotated.

Handling "wrong but confident."

Reliability and cost aredesign inputs, not afterthoughts.

An agent is a distributed systemwith a confident, costly dependencyin the hot path.

Tonight was the map.The cohort is where you build it.

System design for AI agents.
For engineers.

Senior engineers are shipping
agent systems on instinct.

RAG: give it
the facts.

Tool-calling agent:
reason, act, repeat.

Routing:
pick the right path.

Orchestrator
and workers.

An agent is a service that calls
a slow, expensive, flaky dependency.

A normal service versus
an LLM in the path.

State is the
context window.

Four calls decide
whether it works in production.

Reliability and cost are
design inputs, not afterthoughts.

An agent is a distributed system
with a confident, costly dependency
in the hot path.

Tonight was the map.
The cohort is where you build it.