Braintrust

What it is

Evals as tracked experiments.

Braintrust is a managed platform for eval-driven development. It gives you datasets, scorers, experiment tracking, a prompt playground, and a UI to compare runs against each other. It folds evals together with observability, so the failing traces you capture in production and the experiments you run to fix them live in one place, visible to engineers and non-engineers alike.

The one job

Run and compare evals as tracked experiments with a team UI.

Every eval run becomes a versioned experiment you can diff against the last one. That history is what turns "it feels better" into "run 14 beat run 13 on these scorers."

Reach for it when

• You want a managed eval workflow with datasets and experiment history, not scripts.
• You need non-engineers reviewing results in a shared UI.
• You want eval-driven iteration: change the prompt, run the experiment, compare scorers.
• You want evals and observability together so failures feed straight into datasets.

Skip it when

• You want open source or self-host: Langfuse or Phoenix keep it in your own infra.
• You want code or CLI only: DeepEval or Promptfoo stay closer to your repo.
• You are on a tight budget at scale: the hosted model can get expensive as volume grows.

A minimal look

A scorer over a model output.

import { Eval } from "braintrust";
import OpenAI from "openai";
const openai = new OpenAI();

Eval("support-bot", {
  data: () => [{ input: "refund window?", expected: "30 days" }],
  task: async (input) => {                      // call the model under test
    const r = await openai.responses.create({ model: "gpt-5", input });
    return r.output_text;
  },
  scores: [({ output, expected }) => output.includes(expected) ? 1 : 0], // tracked as experiment
});

The principle it teaches

Where it fits your labs

This is a Week 5 tool. When Lab 5 grows from a one-off eval into the flywheel, where captured failures continuously feed the dataset, Braintrust is the platform that keeps the datasets, scorers, and experiment history in one shared place.