Week 3 · Cost, latency & reliability
"Fast enough" and "cheap enough" are numbers. The cost formula, the five levers, caching, routing, buy vs host, the fallback ladder, and the retry trap, on one page.
Before you optimize anything, write the ceiling: this request must cost under $X and answer within Y ms at p95. Without the number you cannot tell a win from a waste.
The demo made one small call. Production hides the bill in the multipliers: agent loops, retries, fat context, long outputs. Prices move, so treat these as illustrative, the ratio is the point.
| Tier | ~Price / million tokens | One 3k-in / 500-out request | At 1M requests / month |
|---|---|---|---|
| Frontier model | $3 in / $15 out | ~$0.017 | ~$17,000 |
| Small model | $0.15 in / $0.60 out | ~$0.0007 | ~$700 |
Do not pay twice for the same work.
Group work; throughput up, cost down.
Answer feels fast: TTFT, not total.
Right model per request.
Smallest model that passes evals.
Prefix caching reuses the tokens of a stable system prompt and context, cheaper on every call. A semantic cache skips the whole model call when a near-identical question already has an answer.
sequenceDiagram
autonumber
participant C as Client
participant K as Cache
participant M as Model
C->>K: request (miss)
K->>M: call model
M-->>K: answer
K-->>C: answer + store
Note over C,K: later, an identical request
C->>K: request (hit)
K-->>C: cached answer, no model call
Most requests are easy. Send them cheap; escalate only the hard ones. Right-sizing is picking the smallest model that still passes your evals, not the smartest you can afford.
flowchart TB
Q["Request"] --> R{"Classify
hard?"}
R -- no --> S["Small model"]
R -- yes --> B["Big model"]
S --> CK{"Confident?"}
CK -- no --> B
CK -- yes --> OUT["Answer"]
B --> OUT
classDef s fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
classDef b fill:#EEE6FF,stroke:#1F2937,color:#0E1726;
class S s; class B b;
Right-sizing picks the model. This picks who runs it. Per-token pricing is rent; a GPU is a mortgage. The crossover is a volume number you can calculate, not a matter of taste.
Pay per token, zero ops, elastic. Wins on low or spiky volume, and on frontier quality you cannot reproduce.
Pay per GPU-hour, busy or idle. Wins on steady high volume, or when the data legally cannot leave.
Hosted cost/req x req/month, versus GPU/month plus the engineer who babysits it.
It will time out, rate-limit, return malformed output, and occasionally go down. That is not an edge case, it is Tuesday.
Timeouts on every call, retries with backoff and jitter, a cap on attempts.
Validate the output; on malformed, retry once then fall back, never crash the request.
When the primary path fails, step down a rung. Every rung is a worse-but-working answer. Users forgive "slightly less smart", they do not forgive an infinite spinner.
flowchart TB A["Big model"] -- "timeout / error" --> B["Small model"] B -- "still failing" --> C["Cached / retrieved answer"] C -- "no cache" --> D["Deterministic default
rules, canned response"] D -- "nothing works" --> E["Honest try-again + log"] classDef ok fill:#D6F5E3,stroke:#1F2937,color:#0E1726; classDef warn fill:#FEF3C7,stroke:#1F2937,color:#0E1726; class A,B ok; class D,E warn;
A retry that re-runs a tool can charge the card twice, send the email twice, book the room twice. Non-determinism makes it worse: the retry may take a different action.
The refund fires twice.
Regardless of how many times you retry the call. Retry the model, not the effect.
| Dial | Target |
|---|---|
| Cost ceiling | < $0.02 typical, < $0.10 worst case (loop + retries) |
| Latency ceiling | < 2s p95, first token < 500ms via streaming |
| Levers on | prefix cache, small-model default, parallel retrieval, stream output |
| On failure | 1 retry with backoff, then small model, then cached, then default; effects idempotent |