Week 5 · Evals & observability
You cannot unit-test a probability. Grade the whole path, gate on it, and grow it from prod. Eval types, graders, metrics, the flywheel, and the trace, on one page.
Traditional tests assert exact outputs. The model returns a distribution, so "assert equals" is dead. Evals replace it: a graded suite that says whether a change made the system better on average, and where it regressed.
Grade only the final answer and you miss the agent that got lucky, burned an expensive tool, or looped five extra times. For agents you grade the trajectory.
Is the final answer correct? Necessary, not sufficient for agents.
Right tools, right args, sane order?
Was the whole path efficient, safe, and generalizable?
flowchart LR G["Task"] --> S1["Step 1
right tool?"] S1 --> S2["Step 2
needed?"] S2 --> S3["Step 3
right args?"] S3 --> A["Answer
correct?"] A --> SC{"Score:
done · efficient · safe"} classDef s fill:#DCEBFE,stroke:#1F2937,color:#0E1726; classDef sc fill:#D6F5E3,stroke:#1F2937,color:#0E1726; class S1,S2,S3 s; class SC sc;
Exact match, schema, "did it call tool X?" Reliable. Trap: only works for checkable facts.
A model scores nuance against a rubric. Trap: biased, needs its own validation against people.
The gold standard for judgment. Trap: slow and costly, so reserve it and sample.
Did it achieve the goal? The headline number.
Right tools, args, order; no needless calls.
A "correct" run that costs 10x is a regression.
Did injection tests get through? Did it over-reach?
Curated cases with known-good outcomes. Run on every change. Your regression gate.
Grade a slice of real production runs continuously. Catches what your golden set never imagined.
The loop that separates teams that improve from teams that thrash. A bad trace is not just an incident, it is a new eval case.
flowchart LR
P["Prod trace fails"] --> C["Capture it"]
C --> E["Add to eval set"]
E --> F["Fix + re-run evals"]
F --> S["Ship, regression locked out"]
S --> P
classDef ok fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
class E,S ok;
{
"input": "refund for BA-2490, you cancelled it",
"expect_tool": "lookup_policy", //trajectory: must check policy
"expect_contains": "carrier-cancellation", //output: cites the clause
"must_not": "state a refund amount not in policy",
"grader": "rule + llm_judge"
}
One request is a trace, each step a span. When a user reports nonsense, open their request id and read this. No guessing.
{ "request_id": "req_8f2a", "outcome": "ok",
"cost_usd": 0.014, "latency_ms": 1830,
"spans": [
{ "type": "retrieve", "ms": 210, "chunks": 5 },
{ "type": "model", "prompt_ver": "v13", "in": 3120, "out": 240 },
{ "type": "tool", "name": "lookup_policy", "ok": true } ] }
| Dashboard | Watch | Alert when |
|---|---|---|
| Cost | spend per request + per day | drifts past the Week 3 budget |
| Latency | p50 / p95 / p99 + first token | the tail crosses your ceiling |
| Quality | online eval pass-rate on sampled traffic | pass-rate drops |