Tuesday decided what to grade. Tonight you build the machine that records it: a trace you can replay, three numbers with lines that page someone, prompt versions that cannot lie, and a human where it counts. You leave with Project 4 and a finished runbook.
Project 4 has six fields. S9 filled two of them. Tonight is the other four, and every one is a decision about your system, not a definition to write down.
When a plane has a bad day, nobody reads the pilot's notes. They pull one box that recorded every input, every instrument, every control movement, in order, for that one flight. They replay it, and they can say what happened without asking anyone who was there.
Most software has the pilot's notes. Somebody printed something useful, somewhere, on a good day. Tonight you build the box: one object per request, holding everything the system saw and did, so that six months from now a stranger can open the worst run of the week and understand it completely, with nobody to ask.
Tuesday you wrote three eval cases and argued about who grades them. That filled two of Project 4's six fields. The other four all sit on something you do not have yet, and cannot fake: a record of what the system actually did.
A log line is a sentence somebody wrote in advance, hoping it would matter later. A trace is a structure the code emits because the code cannot run without emitting it. The difference is not tidiness. It is that you can do arithmetic on one of them and not the other.
The vocabulary is OpenTelemetry's, and borrowing it means your traces land in tooling somebody already built. What matters here is the contents, not the vendor.
| Per request | request id · outcome · total cost · total latency both summed from the spans, never reported separately |
| Per model call | prompt version · tokens in and out · latency · cost a model span costing zero is a bug |
| Per retrieval | the chunk ids so you can re-read exactly what the model was given |
| Per tool call | name · arguments · result · ok or failed without the flag a silent failure is invisible |
| The bar | could a teammate replay and debug this run from the trace alone, with nobody to ask? |
Wrap each step in a span. The span times itself, records what the step produced, and pushes itself onto the trace. Totals are summed from the spans at the end, so the headline cost and the per-step costs can never disagree.
The clock is a parameter. That one decision makes the whole file testable with no API key and no wall-clock flake, and it is why the tests beside it are exact rather than approximate.
export function startTrace(requestId: string, now: Clock): Recorder {
const spans: Span[] = [];
const t0 = now();
return {
async span(type, name, fn, attrs) {
const start = now();
try {
const result = await fn();
const extra = attrs?.(result) ?? {};
// Cost rides on the span, not on a separate meter.
// If it is not here, no dashboard can ever attribute
// spend to a step.
const { costUsd, ...rest } = extra;
spans.push({ type, name, ms: now() - start,
costUsd: costUsd ?? 0, attrs: rest });
return result;
} catch (err) {
// The failing run is the one you most need to
// replay. Record it, then rethrow.
spans.push({ type, name, ms: now() - start, costUsd: 0,
attrs: { failed: true, error: String(err) } });
throw err;
}
},
end(outcome) {
return { requestId, outcome, ms: now() - t0,
// Summed, never reported separately. One source of truth.
costUsd: spans.reduce((s, x) => s + x.costUsd, 0), spans };
},
};
}
"Could a teammate replay this" is a good rule and a useless one, because nobody checks it. So write it down as code: the fields whose absence would force someone to ask you a question. Then it fails in CI instead of at 3am.
It returns the missing fields, not a boolean. A failing trace that says "add a promptVersion to the answer span" is a fix. A red light is a mystery.
export function replayGaps(trace: Trace): string[] {
const gaps: string[] = [];
for (const s of trace.spans) {
if (s.type === "model" && s.attrs.promptVersion === undefined)
gaps.push(`${s.name}: no promptVersion, so you cannot tell
which prompt produced this`);
if (s.type === "model" && s.costUsd === 0)
gaps.push(`${s.name}: costing 0, spend cannot be attributed`);
if (s.type === "tool" && s.attrs.ok === undefined)
gaps.push(`${s.name}: no ok flag, a silent tool failure
is invisible`);
if (s.type === "retrieve" && s.attrs.chunkIds === undefined)
gaps.push(`${s.name}: no chunkIds, you cannot re-read what
the model was given`);
}
return gaps;
}
// And the test that makes it mean something:
it("names every field a debugger would have to ask you for", () => {
const gaps = replayGaps(untracedRun);
expect(gaps).toHaveLength(3);
expect(gaps.join(" ")).toContain("charge_card");
});
S5 told you that you are graded at p95, not at the average. Nobody in this room disagrees. So here is that argument settled by arithmetic instead: one window from the companion repo, 88 requests at 400 ms and 12 at 9000, every number below computed by the same file from the same traces.
Everyone knows not to trust the mean. Almost nobody notices the next step: rolling up percentiles that some other service already computed. Two shards, two p95s, take the average, and you have produced a number with no meaning at all.
If your dashboard reads another dashboard instead of the raw samples, it is not measuring latency. Keep the numbers, not the summaries.
it("the mean of two shard p95s is not the p95 of the traffic", () => {
const quiet = Array.from({ length: 100 }, () => 100);
const busy = [...Array.from({ length: 90 }, () => 200),
...Array.from({ length: 10 }, () => 9_000)];
// What the roll-up dashboard does
const shardAverage =
(percentile(quiet, 95) + percentile(busy, 95)) / 2;
// What the traffic actually did
const truth = percentile([...quiet, ...busy], 95);
expect(shardAverage).toBe(4550);
expect(truth).toBe(200);
// Off by 22x, and it pages someone for a breach
// that never happened.
});
it("refuses to fire on a thin window", () => {
const thin = [trace(400), trace(30_000)];
// p95 of 30s, on two requests, meaning nothing.
// Alerts that fire on noise get muted, and a muted
// alert is worse than no alert.
const w = summarise(thin, { passed: 0, scored: 1 });
expect(evaluateAlerts(w, rules)).toEqual([]);
});
Every alert line you write tonight needs four things: a metric, a direction, a threshold, and a sentence saying why that number. S6 gave you the verb, page or watch. Tonight you give it the number, and the number comes from a percentile over raw samples or it comes from nowhere.
Prompts change weekly and every change moves quality. The moment a trace records a version number that a human typed, the trace is only as honest as the least careful edit anyone made that month. Derive the id from the content and it cannot disagree with what was sent.
Six months later, answer@e7aa2e still returns the exact bytes the model was given. That is the whole feature, and it costs six lines.
// The id is derived, never typed by a human.
export function fingerprint(template: string): string {
return createHash("sha256").update(template)
.digest("hex").slice(0, 6);
}
export function definePrompt(name, template): Prompt {
return { name, template, id: `${name}@${fingerprint(template)}` };
}
// The failure a hand-bumped "v13" gives you:
// edited text, unchanged label, and every trace from
// that day claiming a version that never ran.
it("cannot be edited without the id moving", () => {
const shipped = definePrompt("answer",
"Answer from the policy only.");
const edited = definePrompt("answer",
"Answer from the policy only. Cite it.");
expect(shipped.id).not.toBe(edited.id);
});
Somebody adds four few-shot examples, the golden set goes from 88 to 95, and it ships. Nobody notices that every request now carries those examples forever. S9 ruled that an average going up is a release note, not a decision. Same rule, one file lower.
If your promotion criterion is one number, you are trading two numbers you did not look at.
export function shouldPromote(candidate, incumbent, tolerance) {
const reasons: string[] = [];
if (candidate.passRate < incumbent.passRate + tolerance.minQualityGain)
reasons.push("quality: inside the noise");
if (candidate.costPerRequestUsd >
incumbent.costPerRequestUsd * tolerance.maxCostIncrease)
reasons.push("cost: the longer prompt costs more on every
request, forever");
if (candidate.latencyP95 > incumbent.latencyP95 * tolerance.maxLatencyIncrease)
reasons.push("latency: p95 got worse, and users feel the tail");
return { promote: reasons.length === 0, reasons };
}
// 88% → 95% quality, and it still does not ship:
// { promote: false,
// reasons: [ "cost: the longer prompt costs more on
// every request, forever" ] }
You cannot review everything and you know it. So stop treating human-in-the-loop as a principle and treat it as capacity: a fixed number of runs a day that someone will actually look at. Spend it on stakes first, then on the low-confidence tail.
flowchart LR
R["Agent finished"] --> S{"Irreversible
action?"}
S -- yes --> H["Human reviews"]
S -- no --> C{"Below the
confidence line?"}
C -- yes --> H
C -- no --> AUTO["Ship it"]
H --> LBL["Correction becomes
a permanent case"]
classDef ok fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
classDef h fill:#FEF3C7,stroke:#1F2937,color:#0E1726;
class AUTO ok;
class H,LBL h;
Nobody can defend 0.7. What you can defend is "our reviewers get through five percent of traffic, so the line is wherever the fifth percentile of confidence falls." Sort the sample, cut at capacity, and the threshold becomes an operational fact instead of a preference.
The review queue that nobody drains. It fills because the threshold was a preference, not a capacity, and the moment it is a week deep everything gets rubber-stamped. And the quieter one: a correction that does not become a test case is a favour you did one customer, once.
export function routeForReview(trace, confidence, policy): Decision {
const irreversible = trace.spans.find((s) =>
s.type === "tool" && policy.irreversibleTools.includes(s.name));
// Stakes beat confidence. Confidence is not authority.
if (irreversible)
return { route: "review", reason: `irreversible: ${irreversible.name}` };
if (confidence < policy.minConfidence)
return { route: "review", reason: `low confidence` };
return { route: "auto", reason: "reversible and confident" };
}
// The line comes from capacity, not from a feeling.
export function calibrateThreshold(sample, reviewCapacity) {
const sorted = [...sample].sort((a, b) => a - b);
const budget = Math.floor(sample.length * reviewCapacity);
return sorted[budget];
}
// And the loop that pays for the reviewers:
correctionToCase(trace, input, corrected)
// → { name: "correction-req_8f2a", source: "human-correction",
// traceId: "req_8f2a" } ← the run is still there to replay
This is the depth I am marking against. Notice that not one number in the right-hand column was invented tonight: every single one is carried forward from a decision you already made in an earlier week.
If you cannot name where a threshold came from, it did not come from anywhere. That is the difference between a plan and a wish.
| Field | Decided | Where the number came from |
|---|---|---|
| Trace spec | requestId, outcome, cost, ms, spans[] | replayGaps returns empty on a live run |
| Cost line | > $0.02 / request → watch | the Week 3 per-request budget |
| Latency line | p95 > 3000 ms → page | Project 2's deadline split gave the model 3s |
| Quality line | pass-rate < 0.90 → page | S9's online sampled set |
| Min window | 30 requests | below it any percentile is noise |
| Versioning | name@sha256(template)[0:6] | derived, so a trace cannot lie |
| Promotion | +2% quality, ≤10% cost, ≤10% p95 | S9: an average going up is not a decision |
| Human stays | issue_refund, delete_account, + bottom 5% | irreversible list from the S8 threat model |
| Runbook | 4 entries | the top failure modes from Project 2 |
You already know the shape of an entry, and S6's own caption told you why the silent one was hard: "the symptom line names a metric you had to build." Tonight is the night you build it. So the work is not writing entries again. It is binding every symptom line you already wrote to a field that now exists.
| The symptom line you wrote in S6 | The field in tonight's trace spec that produces it |
|---|---|
| "the latency dashboard p95 crosses the alert line for 5 minutes" | percentile(spans.map(ms), 95) over a ≥30-request window |
| "zero-hit rate above 5 percent" | retrieve spans where chunkIds is empty |
| "citation coverage falls below 80 percent" | sampled pass-rate, graded on the S9 online set |
| "did context bloat?" | model span tokensIn, grouped by promptVersion |
| "users report the assistant hanging" | nothing. This is not a symptom line, it is a customer doing your monitoring for you |
Twenty-six minutes, starting now. I will stay on the call with my microphone muted and I will be watching the chat. Drop your trace spec in the chat when step one is done, and I will read two of them out.
| If you are stuck on | Do this instead |
|---|---|
| "I don't have production traffic, so I have no numbers" | Write the line anyway and mark it provisional, review after 1 week of traffic. A provisional line is a decision. A blank is not. |
| "My system has no tools, so half the spec doesn't apply" | Then say so on the page, in one line. A spec that explains an omission is complete. A spec that silently skips it is not. |
| "I can't pick a confidence threshold" | Skip the number. Write how many runs a day a person could review, and derive it later. Capacity is a fact you already know. |
| "Which failure modes go in the runbook?" | Open Project 2 and take the top-right of your risk matrix. Four entries. Not nine. |
Not because this one is right for your system, but because it is the level of specificity that makes a plan operable. Read your alert lines next to these and ask which of the two a stranger could implement.
| Decorative | Operable | |
|---|---|---|
| Trace spec | "we log requests and responses" | a field list, and every runbook detect line names one of them |
| Alert line | "alert if latency is high" | metric, direction, number, verb, and the sentence saying where the number came from |
| Latency | average response time | a percentile over raw samples, with a minimum window |
| Versioning | "we version our prompts" | how the id is derived, and what a new version has to beat to ship |
| Human-in-the-loop | "a human reviews edge cases" | a named list of always-review actions, a threshold tied to review capacity, and where corrections go |
| Runbook entry | "investigate and fix" | detect, page or watch, first move, then, degrade. Each one a lookup, not a judgement |
The harness plan plus your completed runbook, in the course thread. It is a plan, not a build, but a precise one: someone else could stand up the observability from it, and operate the system from the runbook.
A trace you can replay, because the bar is executable and not a slogan. Three lines with numbers you can source, computed over raw samples, on a window wide enough to mean something. Prompt versions that cannot lie because nobody types them. A human on the irreversible and the uncertain, and every correction becoming a permanent case. And the runbook, finally closed, because you can see the failures now.