Grade the journey, not just the answer.

A right answer reached the wrong way will fail on the next input. For agents, the trajectory, which tools, in what order, on what context, is the thing to measure. Tonight: how to evaluate a non-deterministic system on purpose.

The run sheet.

You'll be able to…

You can't unit-test a probability.

Traditional tests assert exact outputs. The model returns a distribution, so "assert equals" doesn't apply. Evals replace it: a graded suite that tells you whether a change made the system better on average, and where it regressed. No evals means shipping on vibes.

The answer can be right for the wrong reasons.

Grade only the final answer and you miss the agent that got lucky, called an expensive tool it didn't need, looped five extra times, or reached the answer via a path that won't generalize. For agents you grade the trajectory.

Score every step, not just the last.

flowchart LR
  G["Task"] --> S1["Step 1
right tool?"]
  S1 --> S2["Step 2
needed?"]
  S2 --> S3["Step 3
right args?"]
  S3 --> A["Answer
correct?"]
  A --> SC{"Score:
done · efficient · safe"}
  classDef s fill:#DCEBFE,stroke:#1F2937,color:#0E1726;
  classDef sc fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
  class S1,S2,S3 s;
  class SC sc;
        

Four dials on every eval run.

The eval Air Canada didn't run.

Jake Moffatt asked Air Canada's support bot about bereavement fares. It told him he could book now and claim the lower rate afterwards; the airline's actual policy said the opposite. The airline argued the bot was a separate entity responsible for its own words. The tribunal disagreed: negligent misrepresentation, CAD 650.88 in damages.

// one eval case · airline support assistant
{
  "input":           "my grandmother died, can I claim the bereavement rate after I fly?",
  "expect_tool":     "lookup_policy",        // trajectory: read the policy, do not recall it
  "expect_contains": "before travel",        // output: the clause that actually applies
  "must_not":       "promise a retroactive refund",  // the sentence that cost 650.88
  "grader":         "rule + llm_judge"
}

Three graders, each with a trap.

Use rules where you can, LLM-judge where you must, and humans to calibrate the judge. Never trust an LLM judge you haven't checked against people: 94% agreement on a set where 93% of cases pass means the judge learned nothing, because "always say pass" scores 93.

Offline sets gate releases. Online evals watch prod.

You need both. Offline tells you a change is safe to ship; online tells you the real world agrees. And the gate is not "did the average go up": 84% to 91% can still hide four cases that used to pass.

Every failure in prod becomes a permanent test.

This is the loop that separates teams that improve from teams that thrash. A bad trace in production isn't just an incident, it's a new eval case that guarantees the same failure never ships again.

flowchart LR
  P["Prod trace fails"] --> C["Capture it"]
  C --> E["Add to eval set"]
  E --> F["Fix + re-run evals"]
  F --> S["Ship, regression locked out"]
  S --> P
  classDef ok fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
  class E,S ok;
        

Guardrails block at runtime. Evals measure over time.

Write three eval cases.

For your system: one happy-path case, one hard/edge case, and one safety case (an injection attempt from S8). Each with an input and what "pass" means. Drop your safety case in the chat.

Grade the path. Gate on it. Grow it from prod.

Output evals aren't enough, grade the trajectory: completion, tool-calling, cost, safety. Rules where you can, LLM-judge where you must, humans to calibrate. Offline gates releases, online watches prod, and every failure becomes a permanent case.