Lab 3: Add a Budget, Cache & Fallback Ladder

What you'll build

A wrapper that turns a raw call into a reliable one.

One call(request) that knows what it is allowed to spend, answers instantly on a repeat, degrades instead of throwing when the model misbehaves, and never double-charges on a retry.

The shape

Cache, budget, and a ladder for when it breaks.

flowchart TB
  A["Big model"] -- "timeout / error" --> B["Small model"]
  B -- "still failing" --> C["Cached / retrieved answer"]
  C -- "no cache" --> D["Deterministic default"]
  D -- "nothing works" --> E["Honest try-again + log"]
  classDef ok fill:#D6F5E3,stroke:#1F2937,color:#0E1726;
  classDef warn fill:#FEF3C7,stroke:#1F2937,color:#0E1726;
  class A,B ok; class D,E warn;
    

Starter code

Wrap the call.

import OpenAI from "openai";
const client = new OpenAI();
const ladder = ["gpt-4o-mini", "gpt-4o"];       // ladder: cheap first, then capable

async function call(request, budgetUsd = 0.02, timeoutMs = 4000) {
    const key = semanticKey(request);
    const hit = cache.get(key);              // semantic cache: skip the model entirely
    if (hit) return hit;

    let spent = 0.0;
    for (const model of ladder) {
        try {
            const out = await client.chat.completions.create(
                { model, messages: [{ role: "user", content: request }] },
                { timeout: timeoutMs },
            );
            spent += costOf(out);
            if (isValid(out) && spent <= budgetUsd) {
                cache.set(key, out);
                return { answer: out, spent, path: model };
            }
        } catch (err) {                       // timeout or rate limit: step down a rung
            continue;
        }
    }

    return fallback(request);                 // cached, default, or honest try-again
}

async function doRefund(orderId, idemKey) {   // side effects: safe to retry
    if (ledger.seen(idemKey)) {              // idempotency: run once, ever
        return ledger.result(idemKey);
    }
    const result = await payments.refund(orderId);
    ledger.record(idemKey, result);
    return result;
}

Do this, in order

The steps.

1. 1
   Track cost and latency per callCount tokens in and out, multiply by price, time the call. Log spent and ms on every request.
2. 2
   Set a budget and enforce itA cost ceiling and a timeout. If a call would blow the budget, do not make it, degrade.
3. 3
   Add a semantic cacheKey on a normalized/embedded form of the request with a similarity floor. A hit returns instantly, no model call.
4. 4
   Build the fallback ladderBig model, then small model, then cached/default, then an honest "try again". Each rung wrapped in a timeout.
5. 5
   Add retries with backoffOn a transient error, retry with exponential backoff and jitter, capped. Validate the output shape before returning.
6. 6
   Make a side effect idempotentGive any tool that spends money or sends a message an idempotency key, so a retry cannot fire it twice.

You're done when

Acceptance criteria.

• Every call logs its cost and latency, and you can show the p95.
• A repeat request is a cache hit that makes no model call.
• Killing the primary model still returns a usable answer via the ladder.
• A forced retry on the refund tool fires it exactly once (prove it with a counter).
• A malformed model output triggers a retry then a fallback, never a crash.

Go further

Stretch goals.

Wrapping up

Ship the wrapper + the budget.

Hand in your call() wrapper, your budget numbers, and the proof that a retry does not double-charge. This becomes your Project 2 (Cost & Latency Budget + Runbook).