How you'll catch regressions before users do, and the runbook completed for when something still gets through. The operating layer that makes it a system, not a demo.
What this is
The eyes, ears, and memory of your system.
A plan precise enough that a teammate could stand up the observability from it and operate the system from the runbook. Pairs with Lab 5. This is where the Week 2 runbook seed finally gets finished.
💡 In plain English
Two questions you must be able to answer when it breaks: did my change help or hurt (evals), and what did the model actually see and do (traces). This artifact is your plan to answer both, every time.
Do this, in order
The harness plan.
1
Write your trace specFields per request, per model call, per tool call. Pass the replay test: could a teammate reconstruct the run from the trace alone?
2
Define eval sets and gradersAn offline golden set (incl. a safety case) plus online sampling. Rules where you can, LLM-judge where you must, humans to calibrate.
3
Name three dashboards + alert linesCost, latency (p95/p99), quality (pass-rate). What number pages someone, at what threshold?
4
Decide prompt versioning + human-in-the-loopHow prompts are versioned and tied to evals, and the one place a human stays in the loop.
5
Finish the runbookAdd entries for the top failure modes from Project 2, now that you can observe them.
Fill this in
The plan.
Trace spec
each request records … · each model call …
Eval sets
offline golden set of … cases · online sampling at …%
Graders
rules for … · LLM-judge for … · human for …
Dashboards + alerts
cost > … · p95 > … · pass-rate < …
Versioning + HITL
versioned by … · human stays at …
Runbook
now covers failure modes: … (from Project 2)
How it's marked
Operable vs aspirational.
✓ Operable
The trace passes the replay test: someone else could debug a bad run from it.
Every dashboard has an alert threshold, not just a chart.
The runbook covers the top failures with symptom → check → act.
✕ Aspirational
"We'll add monitoring" with no fields.
Dashboards with no alert line, wallpaper.
An eval set with no safety case, or an LLM-judge never checked against people.
Wrapping up
The harness plan + the finished runbook.
It is a plan, not a build, but precise. The bar: could a stranger replay a bad run and fix it using only your plan? This is a whole section of next week's capstone document.
Deliverable
Trace spec, eval sets, graders, dashboards + alerts, versioning, HITL, completed runbook. Due Sun Aug 16.
submit on Maven
Stuck?
Use the S9 cheat sheet's eval-case and trace JSON as your starting shapes, then adapt to your system.
rewatch S9 + S10
For the instructorHow to teach thisclick to open ▾
The replay test runs through everything: could a teammate open a trace and fully reconstruct what happened. If they'd have to ask the author, the spec is not detailed enough. And every dashboard needs a defended line, a chart with no threshold is wallpaper.
This is the payoff of the whole ops arc: now that they can observe the system, they go back to Project 2's failure modes and write real runbook entries. The runbook finally gets finished, closing the loop that started in Week 3.