A customer emails "the parcel turned up smashed, I want my money back". Our bot reads it, applies the policy, and pays. We will write the naive version first, because it is the one everybody ships. Then we will run it, find the bug you cannot see, and fix it in three code changes.
what you need node 20 · zod · any model client (LangChain optional) what we build 8 steps · 3 of them a model call · about 120 lines what it costs nothing. the repo ships recorded answers, so it runs offline
Every system with a model in it has these two halves, and beginners write them as one blob. Keep them apart from line one. The email is untrusted text a human typed while annoyed. The order record is facts your warehouse and ledger will defend in a meeting.
Those four record fields, deliveredAt, condition, promisedBy, tier, are the only things in the entire system capable of catching a model that is confidently wrong.
// the untrusted half
interface Case {
email: string; // a human, typing fast
order: OrderRecord;
}
// the half that can contradict a model
interface OrderRecord {
totalPence: number;
promisedBy: string;
deliveredAt: string | null; // null = never came
condition: "ok" | "damaged" | "wrong_item";
returnWindowDays: number;
previousRefundsPence: number;
tier: "standard" | "priority";
}This is not a strawman, it is the quickstart. One call, five fields out, validated against a schema, done in an afternoon. It is less code than what we build later, it has fewer tests, and on the happy path it is indistinguishable.
Every field you put in here is something you have decided to believe. Most people have never read theirs that way. Ask of each one: if this were wrong, what would disagree with it?
const Decision = z.object({
decision: z.enum(["refund","deny","escalate"]),
refundPence: z.number(),
explanation: z.string(),
notifyCustomer: z.boolean(),
confidence: z.number(),
});
const bot = model.withStructuredOutput(Decision);
const out = await bot.invoke(`
You are a refunds agent. Apply the policy.
Late delivery: 25% capped at 2000p, or 50%
capped at 4000p for priority customers...
ORDER ${JSON.stringify(order)}
EMAIL ${email}`);
await pay(out.refundPence); // ship it$ npm run compare 24 emails · schema-valid 24 / 24 · parse errors 0 · retries 0 c02 priority customer, delivered 3 days late, total GBP 150.00 policy says GBP 40.00 the bot says GBP 75.00 confidence 0.94 it applied the 50% rate and dropped the 4000p cap 5 of 24 wrong · GBP 208.50 paid out incorrectly · 0 caught
A number the model invented has nothing in your codebase to compare it against, so your validator can only ask whether it is an integer. Replace it with one choice out of a set you wrote, plus quotes copied verbatim out of the email. Now both fields can be argued with.
Out of a model: only ever a choice from a set you wrote, plus text it copied from the input. Everything else is you taking its word for it, forever, silently.
// the six things a refund can be. we wrote these.
const REASON_CODES = ["NEVER_ARRIVED", "DAMAGED",
"WRONG_ITEM", "LATE_DELIVERY", "CHANGED_MIND",
"NOT_ELIGIBLE"] as const;
const Reading = z.object({
reasonCode: z.enum(REASON_CODES),
evidence: z.array(z.string()), // VERBATIM spans
});
const reader = model.withStructuredOutput(Reading);
// the prompt shrinks too, and that is the tell:
// "You do NOT decide anything and you do NOT
// calculate money. Pick the one reason. Copy
// the evidence exactly, character for character."It was sitting in the prompt as English prose. Move it into a function and it becomes ordinary, boring, testable code that gives the same answer twice. A rate, a cap and a tier interacting is exactly the shape a fluent model gets confidently wrong, and it is nine lines here.
Repeat: must it answer the same twice? Authority: does it make a fact true by saying it? Recovery: can you undo it a week later? Pricing fails two of the three.
function refundPence(reason, o, now) {
switch (reason) {
case "NEVER_ARRIVED":
case "WRONG_ITEM":
return o.totalPence;
case "DAMAGED":
return withinWindow(o, now) ? o.totalPence : 0;
case "LATE_DELIVERY": {
if (!wasLate(o)) return 0;
const rate = o.tier === "priority" ? 0.5 : 0.25;
const cap = o.tier === "priority" ? 4000 : 2000;
return Math.min(Math.round(o.totalPence*rate), cap);
}
// ...and never below zero, minus prior refunds
}
}A schema check asks whether the shape is right. This asks whether the claim is true, and only the second one catches a confident wrong answer. Four of our six reason codes have a record field that can disagree with them.
CHANGED_MIND and NOT_ELIGIBLE: nothing you hold can dispute them. Write that in a comment rather than pretending. It is where the model's judgement is genuinely the job.
function checkSeam(reading, c) {
// 1 · every quote must really be in the email
const invented = reading.evidence.filter(
(e) => !norm(c.email).includes(norm(e)));
// 2 · the claim, against the order record
const o = c.order, r = reading.reasonCode;
const contra =
r === "NEVER_ARRIVED" && o.deliveredAt !== null
? `record says delivered ${o.deliveredAt}` :
r === "DAMAGED" && o.condition !== "damaged"
? `record says condition "${o.condition}"` :
undefined; // CHANGED_MIND: nothing can disagree
return [...invented, contra].filter(Boolean);
}Three steps, and only the first one is a model. That ordering is the boundary, whether you write it as three function calls or as a graph. Notice what the model no longer touches: the money, the decision, and the send.
When the check fails, escalate, never deny. Fall back toward the outcome you can undo, not the one that is cheapest to serve. Denying is cheapest and it is how you lose a customer who was right.
async function handle(c, now) {
// 1 · model: read the human. that is all.
const reading = await reader.invoke(c.email);
// 2 · code: is it contradicted by what we hold?
const problems = checkSeam(reading, c);
if (problems.length) return escalate(c, problems);
// 3 · code: the money. never the model.
const pence = refundPence(reading.reasonCode, c.order, now);
if (pence > MAX_PER_DECISION) return escalate(c);
return { decision: pence > 0 ? "refund" : "deny",
refundPence: pence, reasonCode: reading.reasonCode };
}
// LangGraph: same three nodes, addConditionalEdges
// on "check" routing to "escalate" or "price".7 of 24 readings land on CHANGED_MIND or NOT_ELIGIBLE. nothing can dispute those. delete the model entirely and keyword rules refuse 7 of 24 emails outright: "it turned up but it is the wrong colour and I would like my money back." so: shrink that set, cap it (GBP 50 a decision), and staff it. do not pretend it is empty. 1 open the schema you pass to withStructuredOutput. delete the free numbers. 2 move the arithmetic into a function. if a record can compute it, compute it. 3 count what crosses the seam and how much you can check. zero means you built a receipt.