Week 5 · Field guide · Eval testing

DeepEval

Pytest for LLMs: write eval cases as unit tests with metrics like relevancy, faithfulness, and hallucination, then run them in CI.

pytest-styleopen sourcePython14+ metrics

What it is

Evals that feel like pytest.

DeepEval, by Confident AI, is "Pytest for LLMs." You write eval cases as unit tests and attach metrics: answer relevancy, faithfulness, hallucination, task completion, and a dozen more. Because the cases look like the tests your team already writes, they drop straight into an existing suite and run in CI. It covers both component-level metrics and end-to-end task metrics.

The one job

LLM evaluations as familiar unit tests.

It reframes evals as something engineers already know how to run and gate. An assertion either passes its metric threshold or fails the test, and the CI job goes red.

Reach for it when

Your team is Python-first and wants evals to feel like pytest, not a new tool.
You want CI-gated quality: the build fails when a metric drops below threshold.
You need both component and end-to-end metrics, from faithfulness to task completion.
You want a rich metric library without writing every judge from scratch.

Skip it when

You are on a TypeScript-only stack: DeepEval is Python, so it needs a separate service.
You want config-only, no-code evals: Promptfoo's YAML is a lighter touch.
You want a hosted workflow with datasets and a team UI: Braintrust fits that better.

The principle it teaches

Course principle

Evals as tests, in CI. Regression detection is the deliverable of Lab 5, and DeepEval gets there by making evals indistinguishable from the unit tests already gating your merges. That is Session 9, expressed in a runner your team already trusts.

Where it fits your labs

This is a Week 5 tool. If your stack leans Python, DeepEval is the natural home for the Lab 5 eval suite: cases as tests, metrics as thresholds, failures as red builds.

Use in

Lab 5 to write eval cases as pytest-style tests with relevancy, faithfulness, and task metrics.

week 5

Pairs with

A CI pipeline that fails the build on a dropped pass rate.