There's a task in something you shipped that goes to a frontier API because that's the reflex, not because the task required it. A quantized 4-bit model on the machine often owns it outright. Then it walks off a cliff, and the only question that matters is whether you find that edge in a rehearsal or in production.
Each one pays a network round trip, a per-request cost, and a privacy surface nobody asked for. Not because the task demanded it. Because reaching for the API is the reflex, and the reflex was formed back when the small models were genuinely useless.
A 3B model at 4-bit is 2.4GB on disk and runs on the laptop you already have. The question stopped being "can it" some time ago. It's now "for which tasks, and how would you know".
flowchart TB t["A small task
classify · extract · tidy"]:::in --> reflex{{"the reflex"}}:::model reflex --> api["Frontier API call"]:::warn api --> a["Network round trip"]:::bad api --> b["Per-request cost
forever"]:::bad api --> c["Your data on
someone's server"]:::bad a --> q["Was any of that
required by the task?"]:::boom b --> q c --> q classDef in fill:#DCEBFE,stroke:#0D1B33,color:#0D1B33; classDef model fill:#EEE6FF,stroke:#0D1B33,color:#0D1B33; classDef warn fill:#FEF3C7,stroke:#0D1B33,color:#0D1B33; classDef bad fill:#FEE4E2,stroke:#0D1B33,color:#0D1B33; classDef boom fill:#0D1B33,stroke:#0D1B33,color:#fff;
The right one is how many dependent reasoning steps does this task need. Answer that and you know which capability tier you need. Only then does where it runs become a question, and at that point it's a straightforward trade on cost and privacy.
Decide capability first, location second. Almost everyone does it the other way round, picks a deployment story, and then discovers the model can't do the job.
Latency sensitivity, privacy exposure and call volume all push work towards the edge. Dependent reasoning steps pushes it back, and it wins, because no amount of cost pressure buys a small model a capability it doesn't have.
"Must work offline" isn't a weight, it's a constraint. No score beats an API you can't reach. If a task must run offline and needs deep reasoning, the honest output is a conflict, not an answer: decompose the task or ship a bigger local model.
$ npm run score -- --latency=high --privacy=high \
--volume=1000000 --steps=1
LOCAL (score 7.0; >=3 local, <=0 frontier)
· latency high: a round trip you cannot amortise
· privacy high: data leaving the device is the whole cost
· 1,000,000 calls: per-request cost dominates at this volume
$ npm run score -- --privacy=high --steps=5 --offline
LOCAL REQUIRED, BUT TOO DEEP. Decompose the task into
single-step pieces, or ship a bigger local model. Do not
paper over it with a better prompt.Pin the tag. qwen2.5 without :3b follows whatever you last pulled, which is how a rehearsed demo becomes a different demo in front of an audience.
ollama ps must say 100% GPU. On CPU you'll see roughly a third of the throughput, conclude local models are hopeless, and be wrong: that's a conclusion about your configuration, not about the model.
$ brew install ollama && ollama serve
$ ollama pull qwen2.5:3b # 2.4GB, Q4_K_M
$ ollama ps
NAME SIZE PROCESSOR CONTEXT
qwen2.5:3b 2.4 GB 100% GPU 4096
$ ollama run qwen2.5:3b --verbose "One line on tides."
eval rate: 40.57 tokens/s # ← write yours downBounded, fixed schema, every answer present verbatim in the input. One step. This is the shape a small model owns, and it's the shape most of your API calls actually are.
The task has an automatic checker. I'm not going to squint at the output and tell you it looks good. It passes or it doesn't, and you'll see which.
$ npm run bench
Pull five fields out of a support ticket
─────────────────────────────────────────────────
local 3B (4-bit) PASS all five fields correct
cheap hosted PASS all five fields correct
frontier PASS all five fields correct
{
"order_ref": "NW-88213",
"amount_gbp": 49,
"card_last4": "4471",
"sentiment": "negative",
"wants_refund": true
}| local 3B, 4-bit | cheap hosted | frontier | |
|---|---|---|---|
| model | qwen2.5:3b | gpt-4o-mini | gpt-4o |
| latency | 5856ms | 1566ms | 1173ms |
| tokens/sec | 11.6 | 33.2 | 44.3 |
| cost / 1M calls | $0 | $58 | $960 |
| data leaves the box | no | yes | yes |
On this machine local was 3.7× slower than the cheap hosted model on the same task. Their hardware beats your laptop by more than the round trip costs you. That's the normal result on a laptop, and it's the one that gets left out of the pitch.
Five shipping rules with an explicit precedence order, and one trap: rule 4 overrides rule 2, so the free-shipping waiver doesn't apply to an offshore postcode. The correct answer is £12.00.
Same length, same output contract, same domain, temperature 0. The only variable is the number of dependent steps. If I changed three things at once this would prove nothing.
Rules, ascending precedence:
1. Base £4.00, or £8.00 over 2kg
2. Orders over £50.00 ship free
3. Fragile adds £2.50
4. Offshore postcodes ALWAYS pay base,
plus a £4.00 offshore surcharge
5. Trade accounts get 50% off the surcharge
Order: 3.4kg · £62.00 · not fragile
IV51 9XR · personal account
→ £8.00 base (rule 1)
→ rule 4 outranks rule 2
→ + £4.00 surcharge
────────────────────
TOTAL: 12.00It finds the first matching rule, applies it, and stops. Clean reasoning, well-formatted, numbered steps, entirely wrong. This is not a prompt problem and a better prompt doesn't fix it.
A model that failed loudly would be fine, you'd catch it. This one produces exactly the shape of output your parser expects, with a wrong number in it. That ships.
local 3B (4-bit) FAIL answered 0.00: took rule 2 and
stopped, missing that rule 4
outranks it
--- local 3B (4-bit) said ---
1. **Base Shipping**:
- The weight is 3.4kg, more than 2kg.
- Therefore base shipping is GBP 8.00.
2. **Order Value and Free Shipping**:
- The order value is GBP 62.00, over GBP 50.00.
- Orders over GBP 50.00 are free of charge.
- This rule overrides the base shipping cost…gpt-4o got this task right in 5 runs out of 6 at temperature 0. The sixth answered £4.00, with confident reasoning. Temperature 0 stops being a determinism guarantee the moment a request crosses a router that can pick a different backend.
Every response you saw tonight is checked in as a recording. DEMO_REPLAY=1 and the whole thing runs with nothing called, which is also how I'd survive this venue's wifi dying mid-talk.
Step 6 in the runbook: add a third task between the two in difficulty and binary-search for the edge. Where it lands is a property of your task shape, not a number you can read off a benchmark table.
$ git clone github.com/ehsangazar/\
lightning-lesson-on-device-starter
$ npm install
$ DEMO_REPLAY=1 npm run cliff # no network needed
# then, with your own key and your own model:
$ npm run bench
$ npm run cliff
$ npm run score -- --privacy=high --steps=2runbook.md first: nine steps, every command written out.Not a task you'd like to move. One you'd bet on. Give me the task and how many dependent steps it needs, and I'll tell you whether I think it survives the cliff.
"We call gpt-4o to classify support tickets into eight categories. One step, the label set is fixed, roughly 40k a month." That's local, obviously and immediately, and it's currently costing real money to export real customer data.
ollama ps says 100% GPU, and write down your tokens/sec before you trust any latency claim, including mine.Tonight was one decision, made properly, on one small task. Making that decision repeatedly and defensibly across a whole system, where models change under you and "it worked when I tried it" isn't evidence, is an architecture problem. That's what the cohort is built on.