Lightning Lesson · 30 min · live

You know your context window's size.
You've never counted what it costs you per turn.

Your tool definitions are re-sent on every single request, before the model does anything you asked for. Tonight we price a real setup on screen, then watch one added tool re-price an entire conversation, because tools render at position zero. Nothing errors. The only symptom is the bill.

Ehsan Gazar
Staff Software Engineer · 16 years in production · 500+ mentees
Where we're going · three ideas, 30 minutes

One argument, in three moves.

0–91 · Count the fixed partthe tokens you re-pay every turn, before any work happens
9–202 · Watch it breakone tool, appended at the end, kills the whole conversation's cache
20–303 · Decide what to throw awaythree eviction policies, judged on what they lose
Everything runs live and offline. The repo needs no API key and makes no network call, so you can clone it on the train home.
By the end of tonight

You'll be able to…

1Measure what your tool definitions really cost. The tokens your MCP servers and schemas consume on every request, before any work happens, with the one command that measures it properly.
2Design an eviction policy for a long trajectory. Compaction, clearing and retrieval, chosen against what each one loses and when it runs, rather than by whichever one was easiest to switch on.
3Protect the cache prefix that pays for itself. What invalidates it, what that costs per turn, and how to order context so it holds.
The recognition beat

Everybody knows the window size.
Nobody knows the standing charge.

Ask an engineer how big their context window is and you get an instant answer. Ask what their agent spends before the user has typed anything and you get a pause, then a guess.

What gets tracked

Window size. Output tokens. Maybe a monthly bill, looked at when it jumps.

What actually bills you

The tool schemas, the system prompt and the history, re-sent in full on every turn. It is not a one-off. It is a per-turn standing charge that grows every time somebody wires in another server.

Nobody decides to spend a tenth of their window on tool definitions. You wire in four servers over a quarter and arrive there.
The reframe · the sentence the rest of this hangs on

A context window is not a size.
It is a budget you re-pay every turn.

Treat it as a size and the only question is "will it fit". Treat it as a per-turn budget and three better questions appear: what am I re-paying for, at what rate, and what do I throw away when it fills. Those are tonight.

The frame · three tiers, and the order is the whole game

Your request renders in a fixed order.

tools
→
system
→
messages
TierChanges how often?What it costs to touch
toolsrarely, and never on purpose mid-conversationeverything after it, which is everything
systemper session, if you interpolate into itthe whole conversation history
messagesevery turn, by designonly what comes after the change
Stable content must physically precede volatile content. Everything that goes wrong tonight is a violation of that one line.
Run 1 · live · the control

Price the part you re-pay.

$ npm run budget

  filesystem      6 tools    3106 bytes  ~ 1195 tokens
  ticketing       4 tools    2874 bytes  ~ 1105 tokens
  docs-search     2 tools    1121 bytes  ~  431 tokens
  system prompt             1958 bytes  ~  490 tokens

  FIXED PREFIX       ~3221 tokens
  ............................  1.6% of a 200K window,
                                spent before the user has said anything.
Three ordinary servers. Twelve tools. The tool definitions cost five times what the system prompt does, and the system prompt is the part people edit.
Before I run the next one

A teammate wires in one more tool.

It is appended to the end of the tools array. It is unrelated to the conversation in flight. It is never called. Your prompt caching is configured correctly and has been reading happily for twenty turns.

A

Nothing. It's one small tool at the end.

B

The tools cache is lost. The rest holds.

C

Everything is lost.

Put A, B or C in the chat. Most rooms say B, and B is the reasonable answer.
Run 2 · live · the slide the talk turns on

Every breakpoint. Not just the tools one.

$ npm run cache

  RUN 2 - ONE TOOL IS ADDED

  DEAD      tools     tool definitions    60faf1235eff -> 85026d8c1e2f
  DEAD      system    system prompt       5c7b254effd7 -> af197e6f0398
  DEAD      messages  turn 2: user        8cbad35e1568 -> 52bf2cc96f26
These are real hashes of the real request prefix, computed in front of you. This is not a rule of thumb about caching. It is your request, answered.
The mechanism · said slowly

"The end of the array" is not the end of the request.

Tools render at position zero. A tool appended to the end of your tools array still lands at the front of what the model sees, ahead of the system prompt and ahead of every message in the conversation.

The cache key at each breakpoint is the bytes of everything before it. Change byte 40 and every key downstream changes with it.

flowchart TB
  T["tools
position zero"]:::hot --> S["system prompt"]:::warm
  S --> M["every message
in the history"]:::warm
  X["one tool appended
to the end of the array"]:::bad --> T
  T -.->|"re-keys everything
downstream"| M
  classDef hot fill:#FEE4E2,stroke:#0D1B33,color:#0D1B33;
  classDef warm fill:#EEE6FF,stroke:#0D1B33,color:#0D1B33;
  classDef bad fill:#0D1B33,stroke:#0D1B33,color:#ffffff;
Which is why the same change made in the messages tier costs nothing: it lands after the cached history rather than before it.
The evidence · same tokens, different rate

What the cache was worth.

TurnsUncachedCachedDifference
1$0.0161$0.0201−$0.0040
10$0.1611$0.0346$0.1264
50$0.8053$0.0990$0.7062
200$3.2210$0.3406$2.8804
Note the first row. Caching costs more if you only ever take one turn, because the write is 1.25×. It pays from turn two, and it is the same tokens billed at a tenth of the rate thereafter.
Where my own advice runs out

Do everything right and it still breaks.

A breakpoint looks back at most twenty content blocks to find a prior cache entry. Exceed that in a single turn and the next request silently misses. No error. No log line.

  blocks appended in one turn      reaches the last entry?
  14                               yes
  20                               yes
  21                               NO - silent miss
Eight parallel tool calls is sixteen blocks before any text. Add images and you are over it having done nothing unusual, in exactly the long agent loop the cache existed to pay for. The fix is a breakpoint every ~15 blocks, not only at the end of the turn.
Zoom out · the other half of the budget

When the window fills, something goes.

Everything so far is about the rate you pay. The rest is about capacity: a long trajectory outgrows the window, and you have three ways to make room. They free almost identical amounts, so that is never the question.

Clear

Delete old tool results outright. Cheapest. Leaves nothing behind.

Compact

Summarise the old turns into one block. Keeps a trace of everything.

Retrieve

Keep nothing, re-fetch on demand. Loses nothing permanently, pays in latency.

The question is what each one destroys, and when it runs.
Run 3 · live · the takeaway

All three lost the same thing.

$ npm run evict   39 blocks, ~37535 tokens, 6 of them load-bearing

  CLEAR TOOL RESULTS   freed ~32400   lost: turn 4, the runbook
  COMPACTION           freed ~32550   lost: turn 4, the runbook
  RETRIEVAL            freed ~36800   lost: turn 4, the runbook

  survived all three: KI-233, the legacy-pipeline finding, the customer's question
The block every policy destroyed was load-bearing evidence sitting inside a large tool result. The findings that survived were the ones restated in a line of their own the moment they were discovered. That is the lever, and it sits upstream of the policy choice.
What to do on Wednesday

Three steps. The first one does the real work.

1Count it. Run count_tokens on the request body you already send, with and without your tools array. The difference is your standing charge. This is the step that changes the conversation, because right now nobody in your team has the number.
2Check one thing you do mid-conversation. Anything that edits system, adds a tool, or interpolates a timestamp into the prefix. Then look at cache_read_input_tokens across repeated requests: if it is zero, you have a silent invalidator.
3Write findings down as you go. One short block per load-bearing fact, the moment your agent discovers it, so that when a policy runs it discards volume rather than evidence.
Yours to keep

Everything you just saw, offline.

Public, MIT, zero runtime dependencies, no API key, no network call. Three commands: budget, cache, evict. Eighteen tests, including one for every claim I made out loud tonight.

  github.com/ehsangazar/lightning-lesson-context-engineering-starter

  npm install && npm run cache   # run 2 is the interesting one
The README carries the caveat this deck carries: byte counts are exact, token counts are estimates and are printed with a ~. Only your provider's tokenizer knows the real number.
Your turn · and I will wait

How many tools does your agent carry?

Not how many it uses. How many are defined in the request. Count the servers you have wired in and multiply.

Tools defined~tokens, at this fixture's rateOf a 200K window
123,2211.6%
307,3303.7%
6014,1707.1%
10023,29011.6%
Put your number in the chat. I'll read them out.
The whole talk · three lines and two corollaries

Recap.

1A context window is not a size. It is a budget you re-pay every turn. The fixed part of that budget is your tool definitions, and almost nobody has counted theirs.
2Tools render at position zero. A tool appended to the end of the array lands at the front of the request, so adding one mid-conversation re-prices the whole conversation. Nothing errors.
3Eviction policies differ in what they destroy, not in what they free. Pick on what you can afford to lose, and when it runs.
+A cache breakpoint looks back twenty blocks. One turn with eight parallel tool calls steps over it, and the miss is silent.
+Evidence inside a large tool result is at the mercy of whichever policy runs. The same fact, restated in one line of your own, survives all of them.
Where to go next

Two things, in this order.

Clone it tonight
github.com/ehsangazar/
lightning-lesson-context-engineering-starter
read runbook.md first
Production-Ready Systems with LLMs and Agents
Four weeks, live. Enrolling now, small and capped.
maven.com/gazar
If your company has a learning budget, this is an expensable line item and I'm happy to send the blurb. Reply and tell me your tool count, I read every one.
Ehsan Gazar
Staff Software Engineer · 500+ mentees · me@gazar.dev