Session 10 · Week 5 · Thu 13 Aug 2026 · 90 min · workshop

Build the eyes and ears.

Tuesday decided what to grade. Tonight you build the machine that records it: a trace you can replay, three numbers with lines that page someone, prompt versions that cannot lie, and a human where it counts. You leave with Project 4 and a finished runbook.

Ehsan Gazar
Production-Ready Systems with LLMs and Agents · session 10 of 12
What you're actually making

Four fields, and the runbook they finally close.

Project 4 has six fields. S9 filled two of them. Tonight is the other four, and every one is a decision about your system, not a definition to write down.

1
Trace spec
what every request records, tested against one bar: could a teammate replay this run without asking you anything.
2
Dashboards + alert lines
cost, latency, quality, each with a threshold you can defend and a verb: page, or watch.
3
Versioning + human-in-the-loop
how a prompt version is decided, and the one place a person stays in the path.
4
The runbook, finished
the entries you could not write in S6, because you could not yet see the failures.
Where we're going · 90 minutes

Forty minutes of me, then you build.

0–13The artefact, and where S9 left youtwo of six fields done
13–22The trace, and the replay testthe foundation everything reads
22–29The trap: what your average is hidingand why p95s never average
29–40Versioning, and where a human staysthe two fields people fudge
40–50One worked harness, then the drillplus the runbook entry
50–76You build yours26 minutes, quiet call
76–90Calibrate, rubric, self-check, submitmark your own work
The working block is protected. If the teaching runs long I cut the worked example, not your 26 minutes. Homework handed out at minute 88 does not get done.
By the end of tonight

Four things you will have done, on your own system.

1Write a trace spec that passes the replay test.Name the fields per request, per model call, per tool call, and prove a stranger could debug from them alone.
2Set three alert lines you can defend out loud.A number, a direction, a verb, and the sentence explaining why that number and not a rounder one.
3Decide how a prompt version is derived, and where a human stays.Two fields, two decisions, both about the system you actually run.
4Close the runbook you opened in S6.Entries for the top failure modes from Project 2, now that you can finally observe them.
Start here · plain English

A flight recorder, not a pile of receipts.

When a plane has a bad day, nobody reads the pilot's notes. They pull one box that recorded every input, every instrument, every control movement, in order, for that one flight. They replay it, and they can say what happened without asking anyone who was there.

Most software has the pilot's notes. Somebody printed something useful, somewhere, on a good day. Tonight you build the box: one object per request, holding everything the system saw and did, so that six months from now a stranger can open the worst run of the week and understand it completely, with nobody to ask.

The test is not "did we log it". The test is "could someone else replay it".
The bridge · S9 gave you the cases

Tonight they stop being JSON and start running.

Tuesday you wrote three eval cases and argued about who grades them. That filled two of Project 4's six fields. The other four all sit on something you do not have yet, and cannot fake: a record of what the system actually did.

Already yours, from S9
Eval sets: how many offline cases, what online sampling rate.
Graders: rules where you can, a judge where you must, a human for the rest.
2 of 6
Tonight, and all four need traces
Trace spec. Dashboards and alert lines. Versioning and human-in-the-loop. The runbook.
Every one of them reads the same object.
4 of 6
The reframe

A log says what you printed. A trace says what happened.

A log line is a sentence somebody wrote in advance, hoping it would matter later. A trace is a structure the code emits because the code cannot run without emitting it. The difference is not tidiness. It is that you can do arithmetic on one of them and not the other.

You cannot compute a p95 from a paragraph. That single sentence is why this session exists.
The trace · what it has to carry

One request is a trace. Every step is a span.

The vocabulary is OpenTelemetry's, and borrowing it means your traces land in tooling somebody already built. What matters here is the contents, not the vendor.

Per requestrequest id · outcome · total cost · total latency both summed from the spans, never reported separately
Per model callprompt version · tokens in and out · latency · cost a model span costing zero is a bug
Per retrievalthe chunk ids so you can re-read exactly what the model was given
Per tool callname · arguments · result · ok or failed without the flag a silent failure is invisible
The barcould a teammate replay and debug this run from the trace alone, with nobody to ask?
If they would have to message you, you did not record enough.
The recorder · in TypeScript

Sixty lines, and the totals cannot drift.

Wrap each step in a span. The span times itself, records what the step produced, and pushes itself onto the trace. Totals are summed from the spans at the end, so the headline cost and the per-step costs can never disagree.

Notice

The clock is a parameter. That one decision makes the whole file testable with no API key and no wall-clock flake, and it is why the tests beside it are exact rather than approximate.

run it → s10/tracing/trace.ts
export function startTrace(requestId: string, now: Clock): Recorder {
  const spans: Span[] = [];
  const t0 = now();

  return {
    async span(type, name, fn, attrs) {
      const start = now();
      try {
        const result = await fn();
        const extra = attrs?.(result) ?? {};
        // Cost rides on the span, not on a separate meter.
        // If it is not here, no dashboard can ever attribute
        // spend to a step.
        const { costUsd, ...rest } = extra;
        spans.push({ type, name, ms: now() - start,
                    costUsd: costUsd ?? 0, attrs: rest });
        return result;
      } catch (err) {
        // The failing run is the one you most need to
        // replay. Record it, then rethrow.
        spans.push({ type, name, ms: now() - start, costUsd: 0,
                    attrs: { failed: true, error: String(err) } });
        throw err;
      }
    },

    end(outcome) {
      return { requestId, outcome, ms: now() - t0,
        // Summed, never reported separately. One source of truth.
        costUsd: spans.reduce((s, x) => s + x.costUsd, 0), spans };
    },
  };
}
The replay test · as a function, not a vibe

Make the bar executable, or it is a slogan.

"Could a teammate replay this" is a good rule and a useless one, because nobody checks it. So write it down as code: the fields whose absence would force someone to ask you a question. Then it fails in CI instead of at 3am.

Notice

It returns the missing fields, not a boolean. A failing trace that says "add a promptVersion to the answer span" is a fix. A red light is a mystery.

the tests → s10/tracing/trace.test.ts
export function replayGaps(trace: Trace): string[] {
  const gaps: string[] = [];

  for (const s of trace.spans) {
    if (s.type === "model" && s.attrs.promptVersion === undefined)
      gaps.push(`${s.name}: no promptVersion, so you cannot tell
                 which prompt produced this`);

    if (s.type === "model" && s.costUsd === 0)
      gaps.push(`${s.name}: costing 0, spend cannot be attributed`);

    if (s.type === "tool" && s.attrs.ok === undefined)
      gaps.push(`${s.name}: no ok flag, a silent tool failure
                 is invisible`);

    if (s.type === "retrieve" && s.attrs.chunkIds === undefined)
      gaps.push(`${s.name}: no chunkIds, you cannot re-read what
                 the model was given`);
  }
  return gaps;
}

// And the test that makes it mean something:
it("names every field a debugger would have to ask you for", () => {
  const gaps = replayGaps(untracedRun);
  expect(gaps).toHaveLength(3);
  expect(gaps.join(" ")).toContain("charge_card");
});
Trap 1 · you already know this one. Here it is, measured

1432 ms says fine. Twelve users in a hundred waited nine seconds.

S5 told you that you are graded at p95, not at the average. Nobody in this room disagrees. So here is that argument settled by arithmetic instead: one window from the companion repo, 88 requests at 400 ms and 12 at 9000, every number below computed by the same file from the same traces.

mean
1432 ms  "about a second and a half, we're fine" p50
400 ms  the median agrees, and both are lying p95
9000 ms  this is the number that pages someone
Twelve percent is not an edge case. It is one user in eight, every day, and none of them will tell you.
The second trap · the one that survives code review

You cannot average two p95s. It is not a smaller error, it is a different number.

Everyone knows not to trust the mean. Almost nobody notices the next step: rolling up percentiles that some other service already computed. Two shards, two p95s, take the average, and you have produced a number with no meaning at all.

The tell

If your dashboard reads another dashboard instead of the raw samples, it is not measuring latency. Keep the numbers, not the summaries.

run it → s10/dashboards/percentiles.test.ts
it("the mean of two shard p95s is not the p95 of the traffic", () => {
  const quiet = Array.from({ length: 100 }, () => 100);
  const busy  = [...Array.from({ length: 90 }, () => 200),
                 ...Array.from({ length: 10 }, () => 9_000)];

  // What the roll-up dashboard does
  const shardAverage =
    (percentile(quiet, 95) + percentile(busy, 95)) / 2;

  // What the traffic actually did
  const truth = percentile([...quiet, ...busy], 95);

  expect(shardAverage).toBe(4550);
  expect(truth).toBe(200);
  // Off by 22x, and it pages someone for a breach
  // that never happened.
});

it("refuses to fire on a thin window", () => {
  const thin = [trace(400), trace(30_000)];
  // p95 of 30s, on two requests, meaning nothing.
  // Alerts that fire on noise get muted, and a muted
  // alert is worse than no alert.
  const w = summarise(thin, { passed: 0, scored: 1 });
  expect(evaluateAlerts(w, rules)).toEqual([]);
});
The fix, in one line

Keep the samples. Alert on the tail.

Every alert line you write tonight needs four things: a metric, a direction, a threshold, and a sentence saying why that number. S6 gave you the verb, page or watch. Tonight you give it the number, and the number comes from a percentile over raw samples or it comes from nowhere.

alert = metric · direction · threshold · because page | watch
A threshold you cannot explain is a threshold somebody will mute in six weeks.
Field 3a · version the prompt, or the trace lies

A hand-typed "v13" is wrong the first time somebody forgets.

Prompts change weekly and every change moves quality. The moment a trace records a version number that a human typed, the trace is only as honest as the least careful edit anyone made that month. Derive the id from the content and it cannot disagree with what was sent.

Notice

Six months later, answer@e7aa2e still returns the exact bytes the model was given. That is the whole feature, and it costs six lines.

run it → s10/versioning/registry.ts
// The id is derived, never typed by a human.
export function fingerprint(template: string): string {
  return createHash("sha256").update(template)
    .digest("hex").slice(0, 6);
}

export function definePrompt(name, template): Prompt {
  return { name, template, id: `${name}@${fingerprint(template)}` };
}

// The failure a hand-bumped "v13" gives you:
// edited text, unchanged label, and every trace from
// that day claiming a version that never ran.
it("cannot be edited without the id moving", () => {
  const shipped = definePrompt("answer",
    "Answer from the policy only.");
  const edited  = definePrompt("answer",
    "Answer from the policy only. Cite it.");
  expect(shipped.id).not.toBe(edited.id);
});
Field 3a · and what gets to ship

The prompt that wins quality and quietly loses cost.

Somebody adds four few-shot examples, the golden set goes from 88 to 95, and it ships. Nobody notices that every request now carries those examples forever. S9 ruled that an average going up is a release note, not a decision. Same rule, one file lower.

The tell

If your promotion criterion is one number, you are trading two numbers you did not look at.

the gate → s10/versioning/registry.test.ts
export function shouldPromote(candidate, incumbent, tolerance) {
  const reasons: string[] = [];

  if (candidate.passRate < incumbent.passRate + tolerance.minQualityGain)
    reasons.push("quality: inside the noise");

  if (candidate.costPerRequestUsd >
      incumbent.costPerRequestUsd * tolerance.maxCostIncrease)
    reasons.push("cost: the longer prompt costs more on every
                  request, forever");

  if (candidate.latencyP95 > incumbent.latencyP95 * tolerance.maxLatencyIncrease)
    reasons.push("latency: p95 got worse, and users feel the tail");

  return { promote: reasons.length === 0, reasons };
}

// 88% → 95% quality, and it still does not ship:
// { promote: false,
//   reasons: [ "cost: the longer prompt costs more on
//               every request, forever" ] }
Field 3b · where a person stays

Human review is a budget, so the only question is which few percent.

You cannot review everything and you know it. So stop treating human-in-the-loop as a principle and treat it as capacity: a fixed number of runs a day that someone will actually look at. Spend it on stakes first, then on the low-confidence tail.

flowchart LR
  R["Agent finished"] --> S{"Irreversible
action?"} S -- yes --> H["Human reviews"] S -- no --> C{"Below the
confidence line?"} C -- yes --> H C -- no --> AUTO["Ship it"] H --> LBL["Correction becomes
a permanent case"] classDef ok fill:#D6F5E3,stroke:#1F2937,color:#0E1726; classDef h fill:#FEF3C7,stroke:#1F2937,color:#0E1726; class AUTO ok; class H,LBL h;
Stakes beat confidence. A confident refund is still a refund.
Field 3b · in TypeScript, and the loop that pays for it

Set the line from your review capacity, not from a round number.

Nobody can defend 0.7. What you can defend is "our reviewers get through five percent of traffic, so the line is wherever the fifth percentile of confidence falls." Sort the sample, cut at capacity, and the threshold becomes an operational fact instead of a preference.

The trap

The review queue that nobody drains. It fills because the threshold was a preference, not a capacity, and the moment it is a week deep everything gets rubber-stamped. And the quieter one: a correction that does not become a test case is a favour you did one customer, once.

run it → s10/hitl/route.ts
export function routeForReview(trace, confidence, policy): Decision {
  const irreversible = trace.spans.find((s) =>
    s.type === "tool" && policy.irreversibleTools.includes(s.name));

  // Stakes beat confidence. Confidence is not authority.
  if (irreversible)
    return { route: "review", reason: `irreversible: ${irreversible.name}` };

  if (confidence < policy.minConfidence)
    return { route: "review", reason: `low confidence` };

  return { route: "auto", reason: "reversible and confident" };
}

// The line comes from capacity, not from a feeling.
export function calibrateThreshold(sample, reviewCapacity) {
  const sorted = [...sample].sort((a, b) => a - b);
  const budget = Math.floor(sample.length * reviewCapacity);
  return sorted[budget];
}

// And the loop that pays for the reviewers:
correctionToCase(trace, input, corrected)
  // → { name: "correction-req_8f2a", source: "human-correction",
  //     traceId: "req_8f2a" }  ← the run is still there to replay
The map to keep · one worked harness, end to end

A billing assistant, all four fields, every number sourced.

This is the depth I am marking against. Notice that not one number in the right-hand column was invented tonight: every single one is carried forward from a decision you already made in an earlier week.

The point

If you cannot name where a threshold came from, it did not come from anywhere. That is the difference between a plan and a wish.

the whole thing runs → s10/demo.ts
FieldDecidedWhere the number came from
Trace specrequestId, outcome, cost, ms, spans[]replayGaps returns empty on a live run
Cost line> $0.02 / request → watchthe Week 3 per-request budget
Latency linep95 > 3000 ms → pageProject 2's deadline split gave the model 3s
Quality linepass-rate < 0.90 → pageS9's online sampled set
Min window30 requestsbelow it any percentile is noise
Versioningname@sha256(template)[0:6]derived, so a trace cannot lie
Promotion+2% quality, ≤10% cost, ≤10% p95S9: an average going up is not a decision
Human staysissue_refund, delete_account, + bottom 5%irreversible list from the S8 threat model
Runbook4 entriesthe top failure modes from Project 2
Field 4 · closing the runbook you opened in S6

S6 named the metric. Tonight you say where it lives.

You already know the shape of an entry, and S6's own caption told you why the silent one was hard: "the symptom line names a metric you had to build." Tonight is the night you build it. So the work is not writing entries again. It is binding every symptom line you already wrote to a field that now exists.

The symptom line you wrote in S6The field in tonight's trace spec that produces it
"the latency dashboard p95 crosses the alert line for 5 minutes"percentile(spans.map(ms), 95) over a ≥30-request window
"zero-hit rate above 5 percent"retrieve spans where chunkIds is empty
"citation coverage falls below 80 percent"sampled pass-rate, graded on the S9 online set
"did context bloat?"model span tokensIn, grouped by promptVersion
"users report the assistant hanging"nothing. This is not a symptom line, it is a customer doing your monitoring for you
Any detect line that does not map to a field is a line you cannot act on at 3am. Delete it or add the field.
Project 4 · capture it here

Eval Harness Plan. Every blank is a decision.

Trace spec
Per request: · per model call: · per tool call:
Eval sets (from S9)
Offline golden set of cases · online sampling at …%
Graders (from S9)
Rules for · judge for · human for
Alert lines
cost > page/watch, because · p95 > , because · pass-rate < , because · min window
Versioning
A prompt version is · it ships when
Human-in-the-loop
Always reviewed: · confidence line , set from capacity · corrections go to
Runbook
Entries for (from Project 2), each with a detect line that names a field in the trace
Build it · in order

The drill. Do not start at the dashboards.

1Write the trace spec first, and run replayGaps against it in your head.Everything else reads this object. Ten minutes. If a field is not here, no later field can use it.
2Three alert lines, each with its "because".Steal the numbers from your Week 3 budget and your Project 2 deadline split. Do not invent new ones tonight.
3Two sentences: how a version is derived, and who a human always reviews.The irreversible list comes straight out of your S8 threat model.
4Close the runbook: fill in the detect line on your top failure modes.Each detect line must name a field that exists in the spec you wrote in step 1. If it does not, go back to step 1.
Workshop · ~26 min

Your system. Four fields. Go.

Twenty-six minutes, starting now. I will stay on the call with my microphone muted and I will be watching the chat. Drop your trace spec in the chat when step one is done, and I will read two of them out.

Your own system
the one you have been carrying since P0, not a hypothetical one
Reuse your numbers
Week 3 budget, P2 deadline split, S8 irreversible list, S9 sampling rate
?
Stuck is normal
unmute and ask. The next slide covers the four places people stall
During the build · the four places people stall

Leave this up. One of these is you.

If you are stuck onDo this instead
"I don't have production traffic, so I have no numbers"Write the line anyway and mark it provisional, review after 1 week of traffic. A provisional line is a decision. A blank is not.
"My system has no tools, so half the spec doesn't apply"Then say so on the page, in one line. A spec that explains an omission is complete. A spec that silently skips it is not.
"I can't pick a confidence threshold"Skip the number. Write how many runs a day a person could review, and derive it later. Capacity is a fact you already know.
"Which failure modes go in the runbook?"Open Project 2 and take the top-right of your risk matrix. Four entries. Not nine.
Calibrate · the same fields, filled

Compare yours to this, honestly.

Not because this one is right for your system, but because it is the level of specificity that makes a plan operable. Read your alert lines next to these and ask which of the two a stranger could implement.

Trace spec
Per request: id, outcome, total cost, total ms, summed from spans. Per model call: promptVersion, tokensIn, tokensOut, ms, costUsd. Per tool: name, args, result, ok. Per retrieval: chunkIds.
Alert lines
cost > $0.02/reqwatch, because that is the Week 3 budget · p95 > 3000 mspage, because P2 gave the model 3s of a 5s deadline · pass-rate < 0.90page · min window 30 requests.
Versioning
Id is name@sha256(template)[0:6], derived at registration. Ships on +2 points of quality with cost and p95 both within 10%.
Human-in-the-loop
Always: issue_refund, delete_account. Plus the bottom 5% by confidence, because two reviewers clear about 40 runs a day. Corrections become permanent eval cases carrying the traceId.
Runbook
4 entries. Each detect line names a field in the spec above: sampled pass-rate < 0.90 AND retrieval spans with < 2 chunkIds.
The rubric · what good looks like

Operable, or decorative.

DecorativeOperable
Trace spec"we log requests and responses"a field list, and every runbook detect line names one of them
Alert line"alert if latency is high"metric, direction, number, verb, and the sentence saying where the number came from
Latencyaverage response timea percentile over raw samples, with a minimum window
Versioning"we version our prompts"how the id is derived, and what a new version has to beat to ship
Human-in-the-loop"a human reviews edge cases"a named list of always-review actions, a threshold tied to review capacity, and where corrections go
Runbook entry"investigate and fix"detect, page or watch, first move, then, degrade. Each one a lookup, not a judgement
Before you submit · seven questions

Mark your own work first.

1Could a stranger replay your worst run from the trace spec alone?
2Does every alert line have a number, a verb, and a because?
3Is any latency number a mean?If yes, it is hiding the users you are about to lose.
4Does any dashboard read another dashboard instead of raw samples?
5Can a prompt be edited without its version changing?
6Is the human-review threshold tied to actual capacity?
7Does every runbook detect line name a field that exists in your trace spec?This is the one that catches the most people. Check it last, check it properly.
How to submit

Project 4, due Sun Aug 16.

The harness plan plus your completed runbook, in the course thread. It is a plan, not a build, but a precise one: someone else could stand up the observability from it, and operate the system from the runbook.

Include
all seven fields of the template, including the two you filled on Tuesday.
The bar
could someone else replay a bad run and fix it, using only your plan?
Carry forward
this becomes a whole section of next week's capstone document.
Recap · then Week 6

Record once. Watch the tail. Version everything. Keep a person on the few.

A trace you can replay, because the bar is executable and not a slogan. Three lines with numbers you can source, computed over raw samples, on a window wide enough to mean something. Prompt versions that cannot lie because nobody types them. A human on the irreversible and the uncertain, and every correction becoming a permanent case. And the runbook, finally closed, because you can see the failures now.

Tuesday · S11 capstone clinic
Everything, on one page. Your four projects were the sections all along: boundary, context, budget, architecture, evals, observability, pulled into one system design document, stressed live in the room.
Tue 18 Aug
Submit Project 4
Eval Harness Plan and the completed runbook, by Sunday.
due Aug 16