Somewhere in the code you shipped, a small job calls a big AI model over the internet. It probably didn't need to. A small model running on your own laptop can often do that same job, for free. But it has a limit, and it goes wrong badly when it reaches it. Better that you find that limit tonight than your users find it next week.
Every one of these sends data across the internet, charges you a small fee, and puts your users' information on someone else's computer. Not because the job needs any of that. Because calling the API is what we all do without thinking, and that habit formed back when small models really were useless.
A small model is now 1.9GB on disk, about the size of a photo album, and it runs on the laptop you already own. "Can it work?" was answered years ago. The real question is "which jobs, and how would I know?"
flowchart TB t["A small job
sort · pull out facts · tidy"]:::in --> reflex{{"the habit"}}:::model reflex --> api["Big model,
over the internet"]:::warn api --> a["A trip across
the network"]:::bad api --> b["A fee, every call,
forever"]:::bad api --> c["Your data on
someone else's computer"]:::bad a --> q["Did the job
actually need any of that?"]:::boom b --> q c --> q classDef in fill:#DCEBFE,stroke:#0D1B33,color:#0D1B33; classDef model fill:#EEE6FF,stroke:#0D1B33,color:#0D1B33; classDef warn fill:#FEF3C7,stroke:#0D1B33,color:#0D1B33; classDef bad fill:#FEE4E2,stroke:#0D1B33,color:#0D1B33; classDef boom fill:#0D1B33,stroke:#0D1B33,color:#fff;
The right one is how many steps of thinking does this job need? One step means the answer is already sitting in front of the model. Three steps means step two only works if step one was right. That number tells you how smart the model has to be. Where it runs is a separate, much easier question you answer afterwards.
Choose how smart first, where it runs second. Nearly everyone does it the other way round: they pick where it runs, then find out the model can't do the job.
Does it need to be fast? Is the data private? How many calls a month? All three push the job onto your own machine. How many steps of thinking pushes it back the other way, and it always wins, because saving money can't make a small model smarter than it is.
"It has to work with no internet" isn't a preference, it's a hard rule. No score beats a server you can't reach. If a job must work offline and needs deep thinking, the honest answer isn't a score, it's "those two don't fit together": break the job into smaller pieces, or use a bigger model on the machine.
$ npm run score -- --latency=high --privacy=high \
--volume=1000000 --steps=1
LOCAL (score 7.0; >=3 local, <=0 big model)
· needs to be fast: you cannot hide a network trip
· private data: sending it out IS the cost
· 1,000,000 calls: a fee per call adds up to real money
$ npm run score -- --privacy=high --steps=5 --offline
MUST RUN LOCAL, BUT IT'S TOO HARD. Break the job into
one-step pieces, or use a bigger model on the machine.
Do not try to fix this with a cleverer prompt.Always write the full name. qwen2.5 without :3b quietly uses whatever you downloaded last, which is how a demo you practised turns into a different demo in front of an audience.
ollama ps must say 100% GPU. If it says CPU you'll get about a third of the speed, decide small models are hopeless, and be wrong. That would be your setup being wrong, not the model.
$ brew install ollama && ollama serve
$ ollama pull qwen2.5:3b # 1.9GB on disk
$ ollama ps
NAME SIZE PROCESSOR CONTEXT
qwen2.5:3b 3.4 GB 100% GPU 32768
$ ollama run qwen2.5:3b --verbose "One line on tides."
eval rate: 40.57 tokens/s # ← speed. write yours downA short, fixed list of fields, and every answer is already written somewhere in the email. Nothing has to be worked out. One step. This is the kind of job a small model does well, and it's the kind most of your API calls actually are.
A script checks the answer. I'm not going to stare at the output and tell you it looks fine. It either passes or it doesn't, and you'll see which.
$ npm run bench
Pull five facts out of a support email
─────────────────────────────────────────────────
small, your laptop PASS all five fields correct
cheap, cloud PASS all five fields correct
top model, cloud PASS all five fields correct
{
"order_ref": "NW-88213",
"amount_gbp": 49,
"card_last4": "4471",
"sentiment": "negative",
"wants_refund": true
}| small, your laptop | cheap, cloud | top model, cloud | |
|---|---|---|---|
| model | qwen2.5:3b | gpt-4o-mini | gpt-4o |
| time per answer | 5.9 sec | 1.6 sec | 1.2 sec |
| words per second | 11.6 | 33.2 | 44.3 |
| cost for 1M calls | $0 | $58 | $960 |
| data leaves your machine | no | yes | yes |
On this laptop the small local model was 3.7× slower than the cheap cloud model doing the same job. Their computers are much faster than your laptop, by more than the trip over the internet costs you. On a laptop that's the normal result, and it's the part people leave out.
Five shipping rules, written in order, where a later rule beats an earlier one. The trap: rule 4 beats rule 2, so "free over £50" does not apply to a remote island postcode. The right answer is £12.00.
Same length, same answer format, same subject, same settings. The only thing I changed is how many steps of thinking it takes. If I changed several things at once, this would prove nothing.
Rules. A LATER rule beats an earlier one:
1. £4.00 to ship, or £8.00 if over 2kg
2. Orders over £50.00 ship free
3. Fragile adds £2.50
4. Offshore (island) postcodes ALWAYS pay
the base price, plus £4.00 extra
5. Trade accounts pay half the £4.00 extra
Order: 3.4kg · £62.00 · not fragile
IV51 9XR · personal account
→ £8.00 base (rule 1)
→ rule 4 beats rule 2
→ + £4.00 offshore charge
────────────────────
TOTAL: 12.00It finds the first rule that matches, uses it, and stops. Neat numbered steps, clear writing, completely wrong answer. This is not a prompt problem, and a cleverer prompt does not fix it.
A model that broke loudly would be fine, you'd spot it in minutes. This one hands back exactly the format your code expects, with the wrong number inside. That goes live.
small, your laptop FAIL said 0.00: used rule 2 and
stopped, never noticed
rule 4 beats it
--- the small model said ---
1. **Base Shipping**:
- The weight is 3.4kg, more than 2kg.
- Therefore base shipping is GBP 8.00.
2. **Order Value and Free Shipping**:
- The order value is GBP 62.00, over GBP 50.00.
- Orders over GBP 50.00 are free of charge.
- This rule overrides the base shipping cost…The top model got this right 5 times out of 6, on the setting that is meant to make it give the same answer every time. The sixth time it said £4.00, and explained itself just as confidently. That setting stops being a promise as soon as your request passes through a service that can send it to a different machine, which most of us now do without realising.
Every answer you saw tonight is saved inside the repo. Set DEMO_REPLAY=1 and the whole thing runs without calling anything, which is also my backup if the wifi here dies mid-talk.
Step 6 of the runbook: add a third job, harder than the first and easier than the second, then keep splitting the difference until you find the exact point it breaks. That point depends on your job, and no benchmark table can tell you where it is.
$ git clone github.com/ehsangazar/\
lightning-lesson-on-device-starter
$ npm install
$ DEMO_REPLAY=1 npm run cliff # no internet needed
# then, with your own key and your own model:
$ npm run bench
$ npm run cliff
$ npm run score -- --privacy=high --steps=2runbook.md first: nine steps, every command written out.Not one you'd like to move. One you'd bet on. Tell me the job and how many steps of thinking it needs, and I'll tell you whether I think a small model survives it.
"We use the top model to sort support emails into eight groups. One step, the list of groups never changes, about 40,000 a month." That one belongs on a laptop, obviously and immediately, and right now it costs real money to send real customer data away.
ollama ps says 100% GPU, and write down your own speed before you believe any claim about speed, mine included.Tonight was one decision, made properly, about one small job. Making that same decision again and again across a whole system, where the models keep changing under you and "it worked when I tried it" isn't proof, is a design problem. That's what the cohort is built on.