Week 5 · Field guide · RAG evals

Ragas

The RAG-specific ruler: it turns "is our retrieval any good?" into numbers like faithfulness, answer relevance, and context precision.

RAG metricsopen sourcePython

What it is

Metrics built for retrieval.

Ragas is an evaluation library specialized for RAG. It computes metrics such as faithfulness (does the answer stay grounded in the retrieved context?), answer relevance, and context precision and recall, using an LLM as the judge. Where a general eval tool scores any output, Ragas knows the shape of a RAG pipeline: question, retrieved chunks, and answer are all first-class inputs, so it can separate a retrieval problem from a generation problem.

The one job

Score a RAG pipeline on faithfulness and relevance.

Point it at your questions, contexts, and answers, and it returns a scorecard. That number is what lets you say whether a retrieval change actually helped, instead of eyeballing a handful of outputs.

Reach for it when

You have a RAG system (your Lab 2 pipeline) and need to measure retrieval and answer quality.
You want to regression-check RAG changes: did a new chunking or reranking step move the metrics?
You need to separate retrieval from generation failures with context precision and recall.
You want RAG-aware metrics out of the box instead of hand-rolling judges.

Skip it when

It is a non-RAG system: a general eval library fits agents or plain generation better.
You need a full CI harness: DeepEval, Promptfoo, or Braintrust give you the test runner and gating.
You are on a TypeScript-only stack: Ragas is Python, so it lives in a separate eval service.

The principle it teaches

Course principle

Evals before optimization. You cannot improve retrieval you do not measure, and RAG fails in two different places, retrieval and generation, that need different fixes. Ragas is the RAG-specific ruler that tells them apart. That is Session 9's lesson applied to Lab 2.

Where it fits your labs

This is a Week 5 tool aimed squarely at Lab 2. Once your RAG pipeline works, Ragas is how you prove it works and defend it against regressions as you tune chunking, retrieval, and reranking.

Use in

Lab 5 to evaluate the Lab 2 RAG pipeline on faithfulness, relevance, and context metrics.

week 5

Pairs with

A tracing tool that samples real production traffic into your Ragas eval set.