Tuesday you learned what to measure. Tonight you build the machine that measures it: tracing, dashboards, prompt versioning, human-in-the-loop. You leave with a harness plan for Lab 5, and your runbook finished.
One artifact powers the rest. Instrument the request once into a trace, and evals, dashboards, alerts, and the flywheel all draw from it. No traces, no observability, that's the S1 "flying blind" failure, fixed.
flowchart LR
REQ["Request"] --> TR[("Trace")]
TR --> EV["Evals"]
TR --> DASH["Dashboards"]
TR --> AL["Alerts"]
EV --> FLY["Flywheel
fails → cases"]
classDef ok fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
classDef m fill:#DCEBFE,stroke:#1F2937,color:#0E1726;
class TR m;
class FLY ok;
Borrow OpenTelemetry thinking: one request is a trace, each step a span. When something goes wrong, you should be able to open the trace and see exactly what the model saw and did, no guessing.
| Per request | input · retrieved context · final output · total cost + latency · outcome |
| Per model call | prompt version · tokens in/out · latency · the raw completion |
| Per tool call | tool name · args · result · success/failure |
| The test | could a teammate replay and debug this run from the trace alone? |
# one request trace · a span tree
{ "request_id": "req_8f2a", "outcome": "ok",
"cost_usd": 0.014, "latency_ms": 1830,
"spans": [
{ "type": "retrieve", "ms": 210, "chunks": 5 },
{ "type": "model", "prompt_ver": "answer@e7aa2e", "in": 3120, "out": 240 },
{ "type": "tool", "name": "lookup_policy", "ok": true } ] }
Cost, latency, prompt version, token counts, every step. When a user reports nonsense, you open their trace and see exactly what happened. That's the replay test, passed.
Cost per request, latency, and the sampled eval pass-rate. Each needs a line you can defend, because a dashboard without an alert line is wallpaper. But look at what the latency one does depending on which number you put on it. This is one real window from the companion repo: 88 requests at 400 ms, 12 at 9000.
Prompts change weekly and every change moves quality. But the moment a trace records a version number that a human typed, the trace is only as honest as the least careful edit anyone made that month. Derive the id from the content and it cannot disagree with what was sent.
// The id is derived, never typed by a human.
export function definePrompt(name, template): Prompt {
const id = `${name}@${createHash("sha256")
.update(template).digest("hex").slice(0, 6)}`;
return { name, template, id }; // → "answer@e7aa2e"
}
// Six months later, that id still returns the exact bytes
// the model was given. That is the whole feature.
it("cannot be edited without the id moving", () => {
expect(definePrompt("answer", "Answer from the policy only.").id)
.not.toBe(
definePrompt("answer", "Answer from the policy only. Cite it.").id);
});Human-in-the-loop isn't failure, it's design. Put a person where the stakes or the uncertainty are highest: approving irreversible actions, labeling ambiguous cases for the eval set, reviewing low-confidence outputs.
flowchart TB
R["Agent result"] --> C{"High stakes
or low confidence?"}
C -- no --> AUTO["Ship automatically"]
C -- yes --> H["Human reviews
approve / correct / label"]
H --> LBL["Correction → eval set"]
classDef ok fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
classDef h fill:#FEF3C7,stroke:#1F2937,color:#0E1726;
class AUTO ok;
class H,LBL h;
Bonus: every human correction is a labeled example. The loop feeds the flywheel.
Nothing to hand in tonight. What you wrote goes straight into Lab 5, the eval harness you actually build, and the observability section of the capstone design document you defend on Thursday.
A trace you can replay, three lines with numbers you can source, prompt versions that cannot lie because nobody types them, and a human on the irreversible and the uncertain. Plus the runbook, finished. That's the operating layer, the difference between a demo and a system.