Week 5 · Field guide · Eval + red-team

Promptfoo

Evals as a CLI you run in CI: declare test cases in YAML, sweep them across prompts and models, and get a comparison matrix plus red-team suites.

regression + red-teamopen-source CLIconfig-driven

What it is

A test suite for prompts.

Promptfoo is a CLI and config-driven eval and red-teaming tool. You declare test cases and assertions in YAML, run them across prompts and models, and get a side-by-side comparison matrix. Its red-team mode adds adversarial and prompt-injection suites out of the box. It was acquired by OpenAI in early 2026, and the open-source CLI keeps shipping, so it stays git-friendly and self-hostable.

The one job

Regression-test and red-team prompts like a CI suite.

It makes prompt quality something you assert and run, not something you spot-check. A dropped assertion fails the run, the same way a broken unit test fails a build.

Reach for it when

You want git-friendly, CI-runnable evals that live next to your code as YAML.
You need to compare prompts and models side by side in one matrix.
You want adversarial and injection test suites, which ties directly to Week 4 security.
You want to catch regressions before users do, gated in the pipeline.

Skip it when

You want a hosted eval UI and workflow: Braintrust or LangSmith fit teams better.
You need deep tracing of production runs: reach for an observability tool instead.
You want code-first assertions only: DeepEval's pytest model may feel more natural.

A minimal look

The suite is a config file.

# promptfooconfig.yaml, checked into git, run in CI
prompts: ["Summarize: {{input}}"]
providers: ["openai:gpt-5", "openai:gpt-5-mini"]
tests:
  - vars: { input: "Our refund policy is 30 days." }
    assert:
      - { type: contains, value: "30" }        # deterministic check
      - { type: llm-rubric, value: "is faithful to the input" }

The principle it teaches

Course principle

Evals are tests. Promptfoo makes them a CLI you run in CI, so quality regressions are caught by the pipeline before they reach users, exactly like a failing unit test blocks a merge. That is Session 9's lesson, made executable.

Where it fits your labs

This is a Week 5 tool with a foot in Week 4. Your Lab 5 eval suite runs from a Promptfoo config, and the same config carries the red-team and injection cases you designed during Week 4 security.

Use in

Lab 5 for regression evals and Week 4 red-teaming with adversarial and injection suites.

week 5 + week 4

Pairs with

Your TypeScript system under test, invoked as the provider from the config.