Week 6 · Field guide · Decision matrix

Which tool, when

The whole field guide on one page. Not a ranking, a decision: every tool mapped to the constraint that should make you reach for it, and nothing before then.

The one rule

Start with the simplest thing that works.

Every tool here earns its place by removing a real, named constraint: a cost you cannot hold, a failure you cannot catch, a scale you cannot serve. Add it then, not before. A tool adopted early is a dependency you now operate, debug, and pay for while it solves a problem you did not have yet.

Course principle

The architecture is the product. Frameworks come and go (Humanloop sunset, AutoGen folded into the Microsoft Agent Framework, Promptfoo acquired). The decisions you make about the code / model boundary, the budget, the threat model, and the eval loop outlive all of them.

The decision path

Ask these in order.

Do you even need a model here?If a rule, a lookup, or plain code answers it, do that. The cheapest, most reliable call is the one you never make.
Workflow or agent?Most tasks want a fixed workflow. Reach for an agent only when the task must decide its own steps at runtime. Start with the five workflow patterns.
Vendor SDK or framework?A single agent on OpenAI wants the OpenAI Agents SDK. Reach for LangGraph or a multi-agent framework only when the control flow outgrows what the SDK can express.
Have you put a number on it?Before you add a gateway or a cache, know your per-request cost and latency budget. Optimize against the number, not a hunch.
Can you catch it when it breaks?No tool ships without tracing and an eval that fails on a regression. Observability and evals are not week-five polish, they are the seatbelt.

The matrix

Reach for it when.

Week 1 · Orchestration & the boundary
Tool	Reach for it when
LangGraph	a multi-step or branching flow needs explicit control, durable state, or human-in-the-loop.
CrewAI	a task genuinely splits into distinct roles and you want a multi-agent crew fast.
OpenAI Agents SDK	a single agent on OpenAI wants tool use, memory, and tracing with no framework tax.
Microsoft Agent Framework	enterprise multi-agent in a .NET or Azure shop.
Pydantic AI	a Python team wants type-safe agents with validated outputs.
Instructor	you need reliable, schema-validated structured output (in TS, Zod does this).
DSPy	you have a metric and want prompts optimized against it, not hand-written.

Week 2 · Context & retrieval
Tool	Reach for it when
LlamaIndex	a document-heavy RAG pipeline with many sources needs connectors and query engines.
Haystack	you want explicit, testable RAG pipelines over ecosystem size.
Vector databases	you need similarity search: start with pgvector, move only when a constraint forces it.
Cohere Rerank	retrieval returns too much noise and you need precision on the top few.

Week 3 · Cost, latency & reliability
Tool	Reach for it when
LiteLLM	you want one OpenAI-shaped API over many providers, with budgets and fallbacks, self-hosted.
Portkey	you want routing, caching, guardrails, and observability in one gateway.
OpenRouter	you want the widest model catalog with zero infrastructure.
Semantic caching	repetitive or overlapping queries let you skip the model on a near-match.
vLLM & Ollama	data must stay in your infra, or volume makes self-hosting cheaper than API calls.

Week 4 · Agent architecture & security
Tool	Reach for it when
NeMo Guardrails	you need structured, auditable input/output/dialog rails declared in config.
Lakera Guard	untrusted input reaches your prompts and you want managed injection and PII detection.
Guardrails AI	you want composable output validators with automatic repair.
Llama Guard & Rebuff	you want open-weight, self-hosted safety and injection classifiers.
MCP	you have multiple tools or data sources to expose to agents through one standard.

Week 5 · Evals & observability
Tool	Reach for it when
Langfuse	you want open-source, self-hosted tracing, prompt versioning, and evals.
LangSmith	you build on LangChain or LangGraph and want the richest native tracing.
Arize Phoenix	you want built-in eval metrics and drift detection out of the box.
Ragas	you need to score a RAG pipeline on faithfulness and relevance.
Promptfoo	you want CI-runnable regression and red-team tests from a config file.
DeepEval	a Python team wants evals that feel like pytest.
Braintrust	you want a managed eval workflow with datasets and experiment history.
Helicone	you want logging and cost dashboards by changing one line.
OpenTelemetry	you want to instrument once and stay backend-agnostic.

Principles that outlive any tool

Four things the churn cannot take.

Start simple.Workflow before agent, code before model, one provider SDK before a framework.
Validate the boundary.Never trust raw model output. Schema-check every input and output, in code.
Evals before optimization.You cannot improve, cache, or route what you do not measure. Build the ruler first.
Do not marry a vendor.A thin seam (a gateway, your own llm.ts) keeps every choice above reversible.