Tuesday was the theory. Tonight you make five concrete decisions about what reaches your model's window, and walk out with Project 1: your Context & Retrieval Design.
flowchart LR
S["1 Sources"] --> C["2 Chunking"]
C --> R["3 Retrieval"]
R --> K["4 Ranking"]
K --> A["5 Assembly"]
A --> M{"Model"}
classDef d fill:#DCEBFE,stroke:#1F2937,color:#0E1726;
classDef m fill:#EEE6FF,stroke:#1F2937,color:#0E1726;
class S,C,R,K,A d;
class M m;
Fill these five boxes and the design is done. The rest of tonight is defending each choice out loud.
| Sources | help-centre articles + resolved tickets · re-index nightly · filter to the user's plan tier |
| Chunking | one article section per chunk, 15 percent overlap · keep title + URL as metadata |
| Retrieval | hybrid: BM25 for error codes + dense for phrasing · top 40 |
| Ranking | cross-encoder re-rank · keep top 5 · drop anything below a score floor |
| Assembly | budget 3k tokens for context · highest score last (near the question) · cite sources |
# retrieval config · support assistant
sources: help_center, resolved_tickets
filter: plan_tier == user.plan # access check BEFORE retrieval
retrieve: hybrid(bm25 + dense), top_k=40
rerank: cross_encoder, keep=5, min_score=0.35
assemble: budget=3000 tokens, best_chunk_last, cite=true
fallback: if max_score < 0.35: say_no_answer() # never invent
Six lines, every decision explicit. If your design can't collapse to something this concrete, a blank is still undecided.
If you cannot answer "could this chunk ever reach a user who should not see it?", the source decision is not finished. Two of tonight's three failure modes are settled right here.
Small chunks match precisely but arrive stripped of context. Large chunks carry context but dilute the match and eat budget. Pick the unit first, then tune size against your own queries.
Metadata filters run first: plan tier, language, date, permissions. They shrink the candidate set and enforce access, so retrieval only ever scores documents this user is allowed to see.
| First pass | hybrid retrieve top 40 · fast and recall-oriented · goal: the right chunk is somewhere in here |
| Re-rank | cross-encoder scores each candidate against the query · slow but accurate · now order actually reflects relevance |
| Keep | survive the top 5 · drop anything under a score floor · a thin, high-signal set beats a fat, noisy one |
Models attend most to the start and end of the window and skim the middle. So fewer, better chunks win, and where you place them in assembly matters as much as which ones survive.
# window budget · support assistant
model_window: 16000 tokens
reserve: system 400 + question 200 + answer 1200
context_budget: 3000 tokens # the rest is headroom, not fuel
order: best_chunk_last # nearest the question, beats lost-in-the-middle
dedup: drop near-identical chunks before counting tokens
cut: lowest-ranked first until under_budget
cite: keep source + url on every chunk that survives
Decide the number, then everything else follows: order for attention, dedup so you do not pay twice, and a rule for what gets cut when it overflows. "Stuff in as much as fits" is not a budget.
Ship the five-decision baseline first and measure it. Add these only where a metric tells you the baseline is losing. Complexity you cannot measure is complexity you cannot defend.
Your "success signal" in the template is one of these, chosen and named. You do not measure it tonight, you commit to it. Week 5 turns the chosen number into a real eval harness.
flowchart LR
subgraph IDX["Indexing: offline, scheduled"]
direction LR
D["Docs"] --> CH["Chunk"] --> EM["Embed"] --> VS[("Vector
store")]
end
subgraph QRY["Query: online, per request"]
direction LR
Q["Question"] --> QE["Embed"] --> RT["Retrieve"] --> RK["Re-rank"] --> AS["Assemble"] --> G{"LLM"} --> AN["Answer"]
end
VS -.-> RT
classDef s fill:#DCEBFE,stroke:#1F2937,color:#0E1726;
classDef m fill:#EEE6FF,stroke:#1F2937,color:#0E1726;
class CH,EM,RT,RK,AS s;
class G m;
The index is built offline and reused. Only the bottom row runs per question. The dotted line is the only place the two phases meet: retrieval reads what indexing wrote.
// ingest.ts: runs offline, on a schedule (nightly re-index)
import { embed, chunk } from "./common/llm";
import { vectors } from "./common/store";
for (const doc of await loadDocs()) { // help articles + tickets
for (const part of chunk(doc, { unit: "section", overlap: 0.15 })) {
const vector = await embed(part.text); // one vector per chunk
await vectors.upsert({
id: part.id,
values: vector,
metadata: { url: doc.url, planTier: doc.planTier, updatedAt: doc.updatedAt },
});
}
}
Chunk, embed, store, with the metadata that later powers access filters and freshness. Run this on a schedule, never per request.
// answer.ts: runs online, once per user question
export async function answer(q: string, user: User) {
const hits = await vectors.search(await embed(q), {
topK: 40,
filter: { planTier: user.planTier }, // access BEFORE retrieval
});
const ranked = await rerank(q, hits); // cross-encoder
const kept = ranked.filter(h => h.score >= 0.35).slice(0, 5);
if (kept.length === 0) return "No confident answer."; // never invent
const context = assemble(kept, { budget: 3000, order: "best-last" });
return complete({
system: "Answer only from context. Cite sources.",
user: q,
context,
});
}
Retrieve, rank, assemble, generate. Every one of tonight's five decisions is a literal argument here: topK, filter, the score floor, the budget, the order.
sequenceDiagram
actor U as User
participant A as App
participant V as Vector store
participant R as Reranker
participant L as LLM
U->>A: Why was I charged twice
A->>A: check plan tier and permissions
A->>V: embed query, search top 40
V-->>A: 40 candidate chunks
A->>R: re-rank against the query
R-->>A: keep top 5 above the floor
A->>L: question plus 5 chunks plus cite rule
L-->>A: grounded answer with sources
A-->>U: answer with citations
Same corpus, same config, one real request. If any hop is undefined in your design, that is the blank you go back and fill.
Every decision names a specific choice and a reason. A reader could build it. Failure paths are explicit.
"We'll use RAG with a vector database." No chunk unit, no ranking, no budget, no failure handling.
Access and freshness are decided at the source, so security and staleness are handled by design.
Retrieval grabs everything, filters permissions later (or never). One prompt away from a leak.
One to two pages, or the filled template plus your pipeline diagram. It doesn't need to be built, it needs to be decided and defensible. This design is the backbone your later artifacts extend.
Five decisions, each with a reason: sources, chunking, retrieval, ranking, assembly. Plus failure paths for ungrounded, stale, and leaked context. That's Project 1.