Session 7, second half · Week 4 · Tue 27 Oct 2026

Build the eyes and ears.

Tuesday you learned what to measure. Tonight you build the machine that measures it: tracing, dashboards, prompt versioning, human-in-the-loop. You leave with Project 4, the Eval Harness Plan, and your runbook finished.

Ehsan Gazar
Production-Ready Systems with LLMs and Agents · session 7 of 8
Tonight builds Project 4

Traces in, decisions out.

0–10The harness, end to endwhat Project 4 is
10–30Tracing: what to recordthe span you can replay
30–50Dashboards + prompt versioningcost, latency, quality
50–75Build your harness plan+ finish the runbook
75–85Human-in-the-loopwhere a person stays
85–90Submit + capstone previewWeek 6 next
By the end of tonight

You'll walk out with…

1A trace spec: exactly what each request records, enough to replay it.
2Three dashboards named: cost, latency, quality, with the alert lines.
3A prompt-versioning and human-in-the-loop plan.
4Project 4 written, and your runbook completed.
The whole machine

Traces feed everything downstream.

One artifact powers the rest. Instrument the request once into a trace, and evals, dashboards, alerts, and the flywheel all draw from it. No traces, no observability, that's the S1 "flying blind" failure, fixed.

flowchart LR
  REQ["Request"] --> TR[("Trace")]
  TR --> EV["Evals"]
  TR --> DASH["Dashboards"]
  TR --> AL["Alerts"]
  EV --> FLY["Flywheel
fails → cases"] classDef ok fill:#D6F5E3,stroke:#1F2937,color:#0E1726; classDef m fill:#DCEBFE,stroke:#1F2937,color:#0E1726; class TR m; class FLY ok;
Tracing · the foundation

Record enough to replay the whole run.

Borrow OpenTelemetry thinking: one request is a trace, each step a span. When something goes wrong, you should be able to open the trace and see exactly what the model saw and did, no guessing.

Per requestinput · retrieved context · final output · total cost + latency · outcome
Per model callprompt version · tokens in/out · latency · the raw completion
Per tool calltool name · args · result · success/failure
The testcould a teammate replay and debug this run from the trace alone?
What a trace actually looks like

One request, replayable.

# one request trace · a span tree
{ "request_id": "req_8f2a", "outcome": "ok",
  "cost_usd": 0.014, "latency_ms": 1830,
  "spans": [
    { "type": "retrieve", "ms": 210, "chunks": 5 },
    { "type": "model", "prompt_ver": "v13", "in": 3120, "out": 240 },
    { "type": "tool", "name": "lookup_policy", "ok": true } ] }

Cost, latency, prompt version, token counts, every step. When a user reports nonsense, you open their trace and see exactly what happened. That's the replay test, passed.

Three dashboards

Cost, latency, quality, each with a line you defend.

$
Cost
spend per request + per day; alert when it drifts past the budget from Week 3.
Latency
p50 / p95 / p99 + first token; alert on the tail, not the mean.
Quality
online eval pass-rate on sampled traffic; alert when it drops.

A dashboard without an alert line is wallpaper. Each one needs the threshold that pages someone.

Treat prompts like code

Version the prompt, or you can't trust the trace.

Prompts change constantly, and each change moves quality. If a trace doesn't record which prompt version produced it, you can't tell whether last night's edit helped or hurt. Version prompts, tie evals to versions, and roll back like any deploy.

Versioned
every prompt has an id; the trace records it; evals run per version.
Compare
v12 vs v13 on the golden set before you promote.
Roll back
a bad prompt is a one-line revert, not an archaeology dig.
Where a person stays

Automate the many. Keep humans on the few.

Human-in-the-loop isn't failure, it's design. Put a person where the stakes or the uncertainty are highest: approving irreversible actions, labeling ambiguous cases for the eval set, reviewing low-confidence outputs.

flowchart TB
  R["Agent result"] --> C{"High stakes
or low confidence?"} C -- no --> AUTO["Ship automatically"] C -- yes --> H["Human reviews
approve / correct / label"] H --> LBL["Correction → eval set"] classDef ok fill:#D6F5E3,stroke:#1F2937,color:#0E1726; classDef h fill:#FEF3C7,stroke:#1F2937,color:#0E1726; class AUTO ok; class H,LBL h;

Bonus: every human correction is a labeled example. The loop feeds the flywheel.

Build it · in order

The harness drill.

1Write your trace spec.List the fields per request, per model call, per tool call. Pass the replay test.
2Name three dashboards + alert lines.Cost, latency, quality. What number pages someone, at what threshold?
3Decide prompt versioning + HITL.How prompts are versioned, and the one place a human stays in the loop.
4Finish the runbook.Add entries for the top failure modes from Project 2, now that you can observe them.
Project 4 · fill this in

Eval Harness Plan.

Trace spec
Each request records: · each model call:
Eval sets
Offline golden set of cases · online sampling at …%
Graders
Rules for · LLM-judge for · human for
Dashboards + alerts
Cost > · p95 > · pass-rate <
Prompt versioning + HITL
Versioned by · human stays at
Runbook
Now covers failure modes: (from Project 2)
How to submit

Project 4, due Sun Aug 16.

The harness plan plus your completed runbook. It's a plan, not a build, but a precise one: a teammate could stand up the observability from it, and operate the system from the runbook.

Include
trace spec, eval sets, graders, dashboards + alerts, versioning, HITL, finished runbook.
The bar
could someone else replay a bad run and fix it using only your plan?
Carry forward
this is a whole section of next week's capstone document.
Recap · then Week 6

Trace once. Watch, alert, and improve from it.

A trace you can replay, three dashboards with alert lines, versioned prompts, and a human where it counts. Plus the runbook, finished. That's the operating layer, the difference between a demo and a system.

Tuesday · S11 capstone clinic
Pull every decision, boundary, context, budget, architecture, evals, into one system design document, with live feedback.
Tue 18 Aug
Submit Project 4
Eval Harness Plan + completed runbook, by Sunday.
due Aug 16