Week 5 · Field guide · Observability + evals

Arize Phoenix

Open-source observability with a deep bench of ready-made eval metrics: trace, score, and detect drift, self-hosted.

tracing + 50+ metricsopen sourcedrift detection

What it is

Tracing plus a metrics library.

Phoenix is the open-source observability tool from Arize, which brings an ML-monitoring heritage. On top of tracing it ships 50+ research-backed eval metrics, including faithfulness, relevance, toxicity, and hallucination, plus drift detection and trace analytics. It is built on OpenTelemetry, so its instrumentation slots into a standard tracing stack rather than a proprietary one.

The one job

Trace and evaluate with metrics that ship with it.

Trace your app and evaluate it against a large library of ready-made metrics, self-hosted, without having to author every grader from scratch.

Reach for it when

You want built-in eval metrics and drift out of the box, not a blank slate.
You want OTel-native tracing that fits a standards-based stack.
You are grading RAG and need faithfulness and relevance scoring ready to go.
You value an ML-monitoring lineage: drift and analytics as first-class views.

Skip it when

You want prompt-management workflows: Langfuse is the better fit there.
You want a managed, LangChain-native tool: reach for LangSmith.
You want the most minimal setup and just one dashboard fast.

A minimal look

Instrument, then the spans flow.

import { OpenAIInstrumentation } from "@arizeai/openinference-instrumentation-openai";
import { registerInstrumentations } from "@opentelemetry/instrumentation";
import OpenAI from "openai";

registerInstrumentations({ instrumentations: [new OpenAIInstrumentation()] });

const openai = new OpenAI();          // calls now emit OpenInference spans
const res = await openai.chat.completions.create({
  model: "gpt-4o", messages,
});                                       // grade it with Phoenix metrics

The principle it teaches

Course principle

Measure trajectories and quality, not just outputs (Session 9). Phoenix ships the metrics that turn that principle into numbers: faithfulness, relevance, toxicity, hallucination, and drift, so you score how the system got there, not only what it returned.

Where it fits your labs

This is a Week 5 tool. When Lab 5 asks you to build graders, Phoenix hands you a metric library so you can start scoring faithfulness and relevance instead of writing every judge yourself.

Use in

Week 5, and Lab 5 graders where its metric library saves you from hand-writing judges.

week 5

Pairs with

OpenTelemetry and OpenInference instrumentation, which feed Phoenix its spans.