Tonight you turn Tuesday's levers into two artifacts for your system: a cost & latency budget and a failure-mode map, and you start the runbook you'll finish in Week 5. That's Project 2.
| Artifact | The question it answers | Who reads it | When it earns its keep |
|---|---|---|---|
| Budget | what may this request cost, in money and in milliseconds? | you, and whoever owns the P&L | design reviews, and the month the invoice triples |
| Failure-mode map | how does each part break, and what happens to the user? | the team, and your reviewer | before launch, and every time you add a hop |
| Runbook | it is breaking right now, what do I do? | whoever is on call, probably not you | 3am, once, and it pays for all of this |
Walk the request and ask at each hop: what happens if this is slow, wrong, or down? That walk is the failure-mode map.
flowchart LR
IN["Input"] --> RET["Retrieval"]
RET --> CTX["Context assembly"]
CTX --> M{"Model call"}
M --> VAL["Validate output"]
VAL --> TOOL["Tool / side effect"]
TOOL --> OUT["Response"]
classDef risk fill:#FEE4E2,stroke:#1F2937,color:#0E1726;
class RET,M,TOOL risk;
The red boxes are where it hurts most: retrieval returns nothing, the model times out or hallucinates, the tool fires twice.
A budget for "the system" is a budget for nothing. Pick the single request that dominates, write its name at the top of the page, and everything else on the page is about that one path.
Budget the one a user waits on. Background jobs have a cost problem; interactive requests have a cost problem and a latency problem, so the interactive one teaches you more.
| Ask | Why it picks the path |
|---|---|
| Which request runs most? | volume multiplies everything, a cheap request run a million times is your invoice |
| Which one does a human wait on? | latency only matters where someone is watching a spinner |
| Which one touches money or data? | blast radius lives here, not in the read-only paths |
| Usually the same one | if all three point at one request, that is your hot path, stop deliberating |
Nobody does at this stage. A budget built on labelled estimates beats no budget, because the estimate is a claim you can be wrong about, and being provably wrong is how the number improves.
Every number carries its source. An estimate you can defend is an engineering artifact; an estimate you can't trace is a wish.
| Rank | Source | Confidence | How to get it in 5 minutes |
|---|---|---|---|
| 1 | Measured | high | one log line from staging, or a single scripted call |
| 2 | Counted | high | tokenise your actual prompt, count the tools and the retrieved chunks |
| 3 | Vendor published | medium | price per million tokens, rate limits, documented p50 latency |
| 4 | Analogous | medium | a comparable feature you already run, adjusted for size |
| 5 | Guessed | low | fine, but write (guess) next to it and give it a date to be replaced |
A support assistant. System prompt and tool schemas are fixed, retrieval pulls four chunks, history is trimmed to the last few turns. Nothing here is exotic; the point is that the total is a sum you can actually write down.
| System + tools | 1,800 tokens fixed every call, and cacheable |
| Retrieved context | 4,000 tokens 4 chunks × ~1,000 |
| History + question | 1,400 tokens |
| Input total | 7,200 tokens → × $3/M = $0.0216 |
| Output | 350 tokens → × $15/M = $0.0053 |
| One call | $0.027 |
| Dominant term | input, 80 percent of it, and 5,800 of those tokens you didn't type |
| So the first lever is | cache the 1,800-token prefix, then shrink the 4,000 |
The per-call number is the easy half. What turns $0.027 into a surprise is the multiplier: the router call, the validation re-ask, the retry, the agent loop that runs until it's satisfied.
Write the expected number of model calls per user action, and the worst case number. If you can't bound the worst case, that's not a budgeting gap, it's a missing loop limit.
| Amplifier | Typical | Worst case | The control |
|---|---|---|---|
| Router / classify call | 1 small call, $0.0005 | same | keep it on the cheap tier, forever |
| Main answer call | 1 × $0.027 | escalates to big model, $0.13 | route on need, not by default |
| Output validation re-ask | 8 percent of requests | 2 re-asks | cap re-asks at 1, then fall back |
| Transport retry | 3 percent | 2 retries, full price each | retry budget, not per-call retries |
| Agent tool loop | 2 round-trips | unbounded without a cap | hard max steps + a spend ceiling per request |
| Per user action | ~$0.030 | ~$0.29 | the two numbers your budget needs |
"Under two cents" is arbitrary until you connect it to what a user pays you. Work backwards from price and usage, and the ceiling stops being a preference you have to defend in a meeting.
"At our price and our usage, this feature costs X percent of revenue, and I'm holding it under Y." That sentence is what makes you the person who owns this feature.
| Price | $20 per seat per month |
| Usage | 130 assistant questions per seat per month (measured, not hoped) |
| At $0.030 | $3.90 per seat → 19.5 percent of revenue |
| Target | keep model spend under 10 percent of revenue |
| So the ceiling is | $0.015 typical per question |
| Which the levers reach | prefix cache halves the input bill, and the small-model default covers the rest |
| Worst case still matters | the $0.29 tail is fine at 1 percent of traffic and fatal at 20 so alert on the mix, not just the mean |
Read the "because" line especially. A ceiling without a reason gets negotiated away by the first person who wants a bigger prompt. A ceiling with arithmetic behind it holds.
No hedging, no "roughly", no ranges wide enough to be unfalsifiable. Where the number is a guess it says so, with a date.
| Request | user asks a question in the support widget |
| Cost ceiling | < $0.015 typical · < $0.29 worst · 10 percent of a $20 seat at 130 questions |
| Latency ceiling | p95 < 2,000 ms · first token < 700 ms |
| Model calls | 1 router + 1 answer typical · hard max 6, then the ladder |
| Dominant cost | input tokens, 80 percent · 5,800 of 7,200 are prompt and retrieval |
| Levers on | prefix cache 1h · small model default · 4 chunks max · stream · 400 max output tokens |
| Confidence | tokens counted · usage measured · re-ask rate is a guess, replace after week 1 of traffic |
"p95 under 2 seconds" tells no hop what to do. Divide the ceiling, give every hop its own deadline, and keep headroom back for the fallback ladder, because the slow path is exactly when you need time to degrade.
Reserve 15 to 25 percent. Spend the whole ceiling on the happy path and your fallback has nowhere to run, so a slow request becomes a failed one.
| Per-hop deadline | each hop is cut at its number, never waited on a timeout is a budget line |
| Headroom buys | one retry, or a drop to the small model, or the cached answer |
| Streaming buys | felt latency, not real latency first token at 1025, done at 1570 |
Borrow the FMEA idea from hardware: for each component, name the failure, its blast radius, and the mitigation. Sort by how much it would hurt.
| Component | Failure | Blast radius | Mitigation |
|---|---|---|---|
| Retrieval | returns nothing relevant | ungrounded answer | "no context, say so" path |
| Model | timeout / rate-limit | request hangs | timeout, retry, fallback ladder |
| Model | confidently wrong | bad answer to user | output validation + evals (wk 5) |
| Tool | retried side effect | double charge | idempotency key |
| Layer | Ways it breaks | The one people forget |
|---|---|---|
| Input | oversized upload, wrong language, empty question, hostile prompt | input that costs money to reject |
| Retrieval | index stale, embedding model changed, zero hits, wrong tenant's docs | a silent tenant leak reads as a good answer |
| Context assembly | window overflow, truncation mid-document, ordering breaks the cache | truncation drops the one relevant chunk |
| Model call | timeout, 429, 5xx, provider outage, deprecation, silent version drift | the same prompt scoring worse after a provider update |
| Output | invalid JSON, refusal, truncation at max tokens, hallucinated citation | valid shape, wrong content, which passes every check you have |
| Tools | wrong args, side effect retried, third-party down, slow tool eats the deadline | a tool that succeeds after you already timed out |
| Agent loop | no progress, oscillation, step cap hit mid-task, spend runaway | the loop that "succeeds" having done nothing |
| Data + privacy | PII into logs or prompts, cross-tenant cache hit, retention breach | the cache key that omits tenant id |
| Cost + capacity | quota exhausted, one tenant burns the month, load spike queues forever | the bug that arrives as an invoice, 30 days late |
Traditional systems fail loudly: a stack trace, a 500, a red dashboard. The expensive LLM failures return 200 OK with a confident paragraph. Your map is thin if every row has an exception behind it.
For each row ask: if this happened right now, in production, would anything on a dashboard move? If the answer is no, the mitigation must include building the signal, not just the fix.
| Silent failure | What the user sees | The signal to build |
|---|---|---|
| Retrieval returns nothing | a fluent, ungrounded answer | hits-returned = 0 rate, and citation coverage |
| Context truncated | an answer missing the key fact | truncation counter per request |
| Cache serving stale answers | yesterday's policy, stated confidently | cache age histogram, TTL alarm |
| Ladder stuck on a low rung | a worse product, working fine | served-by rung as a first-class metric |
| Quality drift after a model update | slowly worse answers | a small eval set on a schedule |
| Wrong tenant's context | someone else's data, plausibly worded | tenant id asserted at retrieval and at cache read |
Everyone's "high impact" means something different, which is why priority arguments go in circles. Write the scale down, score every row against it, and the ordering stops being a matter of opinion.
Reach is how many users. Severity is how bad for each. One user losing money outranks every user seeing a typo, and only an explicit scale gets you that answer reliably.
| Score | Blast radius | Example | Reversible? |
|---|---|---|---|
| 1 | Cosmetic, one request | an awkward sentence, a stray markdown character | yes, instantly |
| 2 | One user, one session | request hangs, they retry and it works | yes |
| 3 | One tenant, sustained | their index is stale, every answer is out of date | yes, with a backfill |
| 4 | All users, feature down | provider outage with no ladder underneath | yes, when they recover |
| 5 | Money, data, or trust | double charge, cross-tenant leak, confidently wrong medical or legal answer | no, and it may be reportable |
Detectability is the LLM twist on the classic FMEA score. A loud failure you catch in seconds is manageable at almost any severity. A silent one you find in a customer email three weeks later is not.
Don't over-model it. The score exists to force the conversation and to sort the table, and the top three rows are the only output that matters.
| Failure | Likely | Blast | Blind | Score | Verdict |
|---|---|---|---|---|---|
| Cross-tenant cache hit | 2 | 5 | 5 | 50 | hard control, this week |
| Retrieval returns nothing | 4 | 3 | 4 | 48 | build the signal, then the fix |
| Confidently wrong answer | 4 | 4 | 3 | 48 | evals in week 5, validation now |
| Tool retried, double effect | 2 | 5 | 2 | 20 | idempotency key, done, closed |
| Model timeout | 4 | 2 | 1 | 8 | ladder already covers it |
| Stray markdown | 3 | 1 | 1 | 3 | accept, write that down |
quadrantChart
title Failure modes by likelihood and blast radius
x-axis Rare --> Frequent
y-axis Low blast radius --> High blast radius
quadrant-1 Fix first
quadrant-2 Guard it
quadrant-3 Accept
quadrant-4 Monitor
"Model timeout": [0.72, 0.5]
"Confidently wrong": [0.62, 0.82]
"Double charge": [0.28, 0.9]
"Tenant leak": [0.2, 0.95]
"Retrieval miss": [0.55, 0.42]
"Stale cache": [0.44, 0.55]
"Cosmetic glitch": [0.5, 0.16]
A cosmetic glitch and a double-charge are not equal. The plot makes the priority argument for you, without a meeting.
Most weak maps have mitigations that are really wishes: "improve the prompt", "monitor closely". Force each row into one of five classes and the vague ones expose themselves immediately.
A documented "we accept this, here's why" is stronger than a mitigation nobody will build. What's not allowed is leaving the cell empty and calling it covered.
| Class | What it does | Example | Costs you |
|---|---|---|---|
| Prevent | the failure cannot happen | tenant id in the cache key, schema-constrained output | design work, up front |
| Detect | you find out fast | zero-hits counter, served-by metric, scheduled evals | a metric and an alert |
| Degrade | a worse answer beats none | fallback ladder, cached answer, deterministic rules rung | a rung to build and to test |
| Recover | undo or re-drive | idempotency key, compensating action, replay from the log | state to keep |
| Accept | documented, not ignored | the occasional awkward sentence | nothing, if you wrote down why |
A failure mode with no signal is a story. Each row you scored highly should end at a number on a dashboard, and a decision about whether it's worth a phone call at 3am.
flowchart LR F["Failure mode
from your map"] --> S["Signal
what moves when it happens"] S --> T["Threshold
the number that means trouble"] T --> D{"Worth waking
a human?"} D -- yes --> P["Page + runbook entry"]:::g D -- no --> W["Dashboard + weekly review"]:::a D -. "no signal exists" .-> B["Build the signal first"]:::w classDef g fill:#D6F5E3,stroke:#1F2937,color:#0E1726; classDef a fill:#DCEBFE,stroke:#1F2937,color:#0E1726; classDef w fill:#FEF3C7,stroke:#1F2937,color:#0E1726;
A runbook entry turns a failure mode into an action. Not "the model is down", but how you'd notice, what you check, what you do, and what stops it happening again. You write one good one tonight; the rest grow from real incidents in Week 5.
Almost every bad entry fails one of these, and all six are fixable in the next twenty minutes. Read the list, then re-read what you wrote.
Hand it to someone who didn't build the system. If they have to ask you a single question to proceed, the entry isn't finished.
| Smell | Looks like | Fix |
|---|---|---|
| The verb is vague | "investigate and resolve" | name the command, the dashboard, the switch |
| Needs the author | "ask Sam which flag" | write the flag name and where it lives |
| No confirmation | act, then hope | state the number that proves it worked |
| Diagnosis before mitigation | root cause at step one | stop the bleeding first, diagnose after |
| Unbounded checks | nine things to look at | three, in order, most likely first |
| Never rehearsed | a switch nobody has flipped | flip it once in staging, then it is real |
| Stuck on | Do this instead |
|---|---|
| "I don't know the token counts" | paste your prompt into any tokeniser, count the retrieved chunks, multiply. Two minutes, and it's a counted number, not a guess. |
| "My system isn't built yet" | budget the design. That's the point: a budget written before the code is a constraint, one written after is an excuse. |
| "My map only has five rows" | go back to the nine layers and do the silent-failure test on each. Thin maps are always missing the quiet failures. |
| "Everything feels high priority" | score all of them 1 to 5 on blast radius alone, then sort. Ties break on blindness. Ranking is a forcing function, not an insight. |
Ceilings are real numbers with arithmetic and a source behind them, typical and worst case.
"Be fast, be reliable." A ceiling with no unit, no worst case, and no reason it's that number.
Failures ranked by blast radius, each with a mitigation class: prevent, detect, degrade, recover, accept.
A flat list where a cosmetic glitch sits next to a double-charge, half of them mitigated by "monitor closely".
The runbook entry could be executed by someone who didn't build it, and it says how you'd know it worked.
A runbook that says "investigate and resolve", or one that only works if you're the person on call.
The budget table, the failure-mode map (ranked, with mitigation classes), and one complete runbook entry. The runbook is intentionally a seed, you'll grow it in Week 5 when real traces surface new failures.
You know your ceilings and where they came from. You know how each layer breaks, including the failures that throw nothing, ranked by what they'd cost you. And you have a runbook entry a stranger could run.