Tuesday you learned what to measure. Tonight you build the machine that measures it: tracing, dashboards, prompt versioning, human-in-the-loop. You leave with Project 4, the Eval Harness Plan, and your runbook finished.
One artifact powers the rest. Instrument the request once into a trace, and evals, dashboards, alerts, and the flywheel all draw from it. No traces, no observability, that's the S1 "flying blind" failure, fixed.
flowchart LR
REQ["Request"] --> TR[("Trace")]
TR --> EV["Evals"]
TR --> DASH["Dashboards"]
TR --> AL["Alerts"]
EV --> FLY["Flywheel
fails → cases"]
classDef ok fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
classDef m fill:#DCEBFE,stroke:#1F2937,color:#0E1726;
class TR m;
class FLY ok;
Borrow OpenTelemetry thinking: one request is a trace, each step a span. When something goes wrong, you should be able to open the trace and see exactly what the model saw and did, no guessing.
| Per request | input · retrieved context · final output · total cost + latency · outcome |
| Per model call | prompt version · tokens in/out · latency · the raw completion |
| Per tool call | tool name · args · result · success/failure |
| The test | could a teammate replay and debug this run from the trace alone? |
# one request trace · a span tree
{ "request_id": "req_8f2a", "outcome": "ok",
"cost_usd": 0.014, "latency_ms": 1830,
"spans": [
{ "type": "retrieve", "ms": 210, "chunks": 5 },
{ "type": "model", "prompt_ver": "v13", "in": 3120, "out": 240 },
{ "type": "tool", "name": "lookup_policy", "ok": true } ] }
Cost, latency, prompt version, token counts, every step. When a user reports nonsense, you open their trace and see exactly what happened. That's the replay test, passed.
A dashboard without an alert line is wallpaper. Each one needs the threshold that pages someone.
Prompts change constantly, and each change moves quality. If a trace doesn't record which prompt version produced it, you can't tell whether last night's edit helped or hurt. Version prompts, tie evals to versions, and roll back like any deploy.
Human-in-the-loop isn't failure, it's design. Put a person where the stakes or the uncertainty are highest: approving irreversible actions, labeling ambiguous cases for the eval set, reviewing low-confidence outputs.
flowchart TB
R["Agent result"] --> C{"High stakes
or low confidence?"}
C -- no --> AUTO["Ship automatically"]
C -- yes --> H["Human reviews
approve / correct / label"]
H --> LBL["Correction → eval set"]
classDef ok fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
classDef h fill:#FEF3C7,stroke:#1F2937,color:#0E1726;
class AUTO ok;
class H,LBL h;
Bonus: every human correction is a labeled example. The loop feeds the flywheel.
The harness plan plus your completed runbook. It's a plan, not a build, but a precise one: a teammate could stand up the observability from it, and operate the system from the runbook.
A trace you can replay, three dashboards with alert lines, versioned prompts, and a human where it counts. Plus the runbook, finished. That's the operating layer, the difference between a demo and a system.