The full tour: golden signals, SLIs, SLOs and error budgets, the pillars (metrics, logs, traces, profiling, events), OpenTelemetry, the tool landscape, burn-rate alerting, incident response, and what "ready to run in production" actually means.
An SLO turns "it should be fast and up" into a number you can defend, alert on, and spend. The error budget is what makes reliability a deliberate trade against velocity instead of an argument. And the telemetry underneath is only worth its cost when it changes a decision.
You can't fix what you can't see, so put gauges on the dashboard. And only sound the alarm when the driver actually feels something wrong, not every time a sensor twitches, or you'll learn to ignore the alarm.
Monitoring is the dashboards you built for failures you already predicted. Observability is being able to ask a question you never anticipated, at 2am, from the data you already emit, without shipping new code. One is a fixed set of gauges; the other is the ability to investigate. Formally: how well you can infer a system's internal state from its outputs.
flowchart LR M(["Monitoring"]) --> MK(["answers questions
you knew to ask"]) O(["Observability"]) --> OK(["answers questions
you did not see coming"]) MK --> EX1(["is CPU over 80?"]) OK --> EX2(["why are only Android users
in Brazil seeing slow checkout?"]) classDef known fill:#DCEBFE,stroke:#1F2937,color:#0E1726; classDef new fill:#D6F5E3,stroke:#1F2937,color:#0E1726; class M,MK,EX1 known; class O,OK,EX2 new;
Monitoring is the warning lights on your dashboard: fixed, pre-decided. Observability is being able to pop the bonnet and trace any wire you like when a light you never installed should have come on.
Google's SRE book distils "what to watch on a user-facing system" to four. If you can only build four graphs, build these. Everything else is diagnosis; these four tell you whether users are being served.
Speed, volume, failures, and fullness. If a doctor could ask a patient only four questions before triage, it would be these. Everything deeper comes after.
Golden signals are the idea; RED and USE are the recipes. Apply RED to every request-driven service and USE to every resource (CPU, disk, pool, queue). Between them you cover "is the service healthy" and "is any resource the bottleneck".
RED asks the shop "how many customers, how many left unhappy, how long did they wait?" USE asks each till "how busy, how long's the queue, any jams?" Two lenses, full coverage.
A Service Level Indicator is a ratio of good events to total events, chosen because it tracks what the user actually feels. Pick the few that matter, latency, error rate, availability, and resist measuring everything. A good SLI moves when users hurt and stays flat when they don't.
Don't measure the temperature of every wire in the building. Measure the one thing the people inside feel: is the room comfortable? An SLI is that one honest reading.
A Service Level Objective is the line in the sand on an SLI, over a window. It is a deliberate choice, not an aspiration: 100% is the wrong target because it forbids all risk and all velocity. Write it as a sentence with numbers.
"99.9% of checkout reads complete in under 200ms, measured over a rolling 28-day window."
"Fast and reliable" is unfalsifiable. "Nine in ten thousand reads may be slow, over a month" is a promise you can check on a Tuesday and argue about with data.
They nest. The SLI is the number you observe. The SLO is the internal target you hold yourself to. The SLA is the promise to a customer with money attached if you miss. Golden rule: your SLO should be stricter than your SLA, so you get an early warning before you owe refunds.
| Term | What it is | Audience | If missed |
|---|---|---|---|
| SLI | the measured indicator (a ratio) | engineers | nothing, it's just the reading |
| SLO | your internal target on the SLI | the team | error budget burns, freeze rule |
| SLA | contractual promise to a customer | customers, legal | credits, refunds, penalties |
SLI is the speedometer reading. SLO is the speed limit you set yourself. SLA is the one written on the road with a fine attached. Set your own limit below the legal one and you rarely pay the fine.
The error budget is 1 − SLO: the amount you are allowed to fail before you must slow down and fix. It converts reliability-vs-velocity from a shouting match into arithmetic. Budget left, ship features. Budget spent, the freeze rule kicks in and reliability work takes priority.
| SLO | Budget / month | Feels like |
|---|---|---|
| 99% | ~7h 18m | generous, internal tools |
| 99.9% | ~43m | typical user-facing service |
| 99.95% | ~21m | important revenue path |
| 99.99% | ~4m | expensive, rarely needed |
It's a spending account for downtime. Each nine you add cuts the monthly allowance by ten. The point isn't to hoard it, unspent budget means you're moving too slowly; the point is to spend it on purpose.
The budget is toothless without a written policy that everyone, including product, agreed to in advance. It removes the argument from the moment of the incident: the decision was made when heads were cool.
Agree the house rules before anyone's losing. "If we overspend the downtime budget, features stop until we're back in the black" is a decision no one can relitigate mid-fire.
The classic three pillars are metrics, logs and traces, each best at one job. Two more are now mainstream: continuous profiling (where the CPU/memory goes inside a process) and wide events (one fat structured record per unit of work). Reach for the wrong one and you either drown in cost or can't answer the question.
Metrics are numbers over time: request rate, error count, p99 latency. They are cheap to store and aggregate, which makes them the home of dashboards and SLOs. The trap is cardinality: tag a metric with something high-variety like user ID, and you multiply into millions of series and a runaway bill.
flowchart LR M(["request latency
one metric"]) --> OK(["tag: region, route
tens of series · fine"]) M --> BAD(["tag: user id, request id
millions of series ·
cardinality blowup"]) classDef good fill:#D6F5E3,stroke:#1F2937,color:#0E1726; classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726; classDef bad fill:#FEE4E2,stroke:#1F2937,color:#0E1726; class M io; class OK good; class BAD bad;
A metric is a tally on a whiteboard. Tally by a handful of categories and it's readable. Start a fresh tally for every single customer and you need a warehouse of whiteboards and still can't read them.
Getting the metric type wrong quietly breaks your dashboards. Counters only go up (rate them). Gauges go up and down. Histograms bucket observations so you can compute percentiles across instances, which summaries can't.
http_requests_total. Query as a rate.queue_depth, temperature, memory in use.A counter is an odometer, only ever climbing, so you read the speed as the change. A gauge is the fuel needle, live and bidirectional. A histogram keeps every lap time so you can ask "what was my 99th slowest lap" later.
A mean latency hides the tail where real pain lives. If p50 is 40ms but p99 is 900ms, one in a hundred requests is miserable, and on a page with 100 assets that's almost every page. Alert and SLO on percentiles (p90, p99, p99.9), never the average.
Put Bill Gates in a room and the average net worth says everyone's a billionaire. The median tells the truth about the room; the tail tells you who's hurting. Latency is the same shape.
A log is a timestamped record of one event, with all the context a metric throws away. That detail is the point and the price: logs are expensive to store and search. Emit them structured (key-value, not prose) so you can query by field, and sample the chatty ones.
An unstructured log is a story you have to read end to end. A structured log is a form with labelled boxes, so you can instantly pull "every checkout that failed on payments last hour" without reading a word.
Three habits separate useful logs from noise. Use levels honestly (ERROR means a human should care). Stamp every line with a correlation / trace ID so you can stitch one request's logs across services. And never log secrets or PII, logs leak and are retained.
Give every request a wristband number at the door. Now every log line it touches, in any service, carries that number, and reconstructing its night out is a single search, not detective work.
A trace stitches one request's journey across every service into a tree of timed spans. When p99 is bad but no single service looks broken, the trace shows you the hop that ate the time, the thing metrics and logs alone can't point to.
flowchart LR G(["gateway
span 12ms"]) --> A(["auth
span 8ms"]) A --> C(["checkout
span 30ms"]) C --> P(["payments
span 540ms"]) C --> D(["DB
span 14ms"]) classDef ok fill:#D6F5E3,stroke:#1F2937,color:#0E1726; classDef slow fill:#FEE4E2,stroke:#1F2937,color:#0E1726; class G,A,C,D ok; class P slow;
Your parcel is late and the tracking page shows every stop with a timestamp. You don't guess; you see it sat in one depot for two days. A trace is that tracking page for a request, and payments is the depot.
A trace only connects if each service passes the trace context onward, the W3C traceparent header carrying trace ID, span ID and flags. And because tracing everything is costly, you sample. The choice of when to decide matters enormously.
Head sampling is deciding to photograph every hundredth car as it enters the tunnel, you'll miss the crash. Tail sampling waits to see which cars crashed, then keeps those photos. More work, far more useful.
This is the view in Jaeger, Tempo or Datadog APM. Each bar is a span; length is time, indentation is the call tree. The eye goes straight to the long red bar, payments, and the gap after it (a retry). No guessing which service to blame.
It's a Gantt chart for a single request. The longest bar is your suspect; a bar that starts only after another finishes tells you what was blocking on what. You read the fault off the picture.
Traces stop at the service boundary. When one service is slow for no external reason, continuous profiling (Pyroscope, Parca, the Go/JVM profilers) samples stacks in production and renders a flame graph. Width is time on CPU; the widest tower is where the cycles go.
A flame graph is a heat map of "where is the CPU spending its day". The widest brick, here a giant JSON serialize, is the hog. You optimise the wide bricks and ignore the thin ones.
The newer school (Honeycomb, and increasingly OpenTelemetry) says: emit one wide, high-cardinality event per unit of work, with dozens of fields (user tier, region, build, feature flags, timings). Then slice by any field after the fact, without having predicted the question. Metrics answer "how many"; wide events answer "which ones, and what did they have in common".
Metrics are tally marks you decided to keep. A wide event is keeping the whole receipt for every transaction, so later you can group by anything on it: card type, store, hour, cashier, without having guessed you'd need to.
OTel is the CNCF standard (a merge of OpenTracing and OpenCensus) for generating traces, metrics and logs. You instrument with vendor-neutral SDKs and auto-instrumentation, ship over the OTLP protocol to a Collector, and the Collector fans out to whatever backend you like. The payoff: your instrumentation no longer marries you to one vendor.
flowchart LR APP(["your app
OTel SDK + auto-instrument"]) -->|OTLP| COL(["OTel Collector
receive · batch · sample · redact"]) COL --> PROM(["Prometheus /
Mimir · metrics"]) COL --> TEMPO(["Tempo / Jaeger ·
traces"]) COL --> LOKI(["Loki / ELK ·
logs"]) COL --> SAAS(["Datadog / Honeycomb ·
if you prefer SaaS"]) classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726; classDef hub fill:#DCEBFE,stroke:#1F2937,color:#0E1726; classDef out fill:#D6F5E3,stroke:#1F2937,color:#0E1726; class APP io; class COL hub; class PROM,TEMPO,LOKI,SAAS out;
OTel is a universal plug. Wire your app to the plug once; swap the wall socket (backend vendor) whenever you like without rewiring the house. That's the whole pitch, and it's why it won.
In a real incident you walk the signals in order, each narrowing the next. A metric alert tells you something is wrong and how much; a trace tells you where; logs tell you why; a profile tells you why inside the slow service.
flowchart LR S(["alert: p99
checkout up"]) --> MET(["METRICS
what + how bad"]) MET --> TR(["TRACES
which hop"]) TR --> LG(["LOGS
why it failed"]) LG --> PR(["PROFILE
why it's slow
inside"]) PR --> FIX(["root cause"]) classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726; classDef step fill:#DCEBFE,stroke:#1F2937,color:#0E1726; classDef good fill:#D6F5E3,stroke:#1F2937,color:#0E1726; class S io; class MET,TR,LG,PR step; class FIX good;
Smoke alarm (metric) says there's a fire and how big. You follow the smoke (trace) to the room, open the door (logs) to see what caught, and inspect the appliance (profile) to see why it sparked. One path, in that order.
Don't assemble twelve tools from scratch. Pick a stack that matches your scale, team size and cloud, then grow into it.
| Stack | What it is | Wins when | Watch out |
|---|---|---|---|
| OSS "LGTM" | Prometheus + Grafana + Loki + Tempo + Alertmanager, fed by OTel | cost-sensitive, own your data, k8s-native | you run it: storage, upgrades, on-call for the tooling |
| SaaS all-in-one | Datadog, New Relic, Honeycomb | small team, want it working today, correlation built-in | cost scales with hosts + cardinality; bill shock |
| Cloud-native | CloudWatch, Google Cloud Ops, Azure Monitor | single-cloud, basics only, zero extra vendors | weaker tracing/UX, poor cross-cloud, lock-in |
Self-host to save money and own the data if you have hands to run it. Pay a SaaS to skip the plumbing if you have budget not time. Use the cloud's built-in if you're small and single-cloud. All three are defensible; picking none and hand-rolling everything is not.
A good service dashboard fits on one screen and answers "are users OK, and how much budget is left" in three seconds. Golden signals across the top, error budget bottom-right. No vanity graphs.
Like a car's dashboard: speed, revs, fuel, temperature, all glanceable, plus a low-fuel light (budget nearly gone). If a panel wouldn't change what you'd do, it doesn't belong on the first screen.
High CPU is not an incident; a user who can't check out is. Page on the SLO being at risk, the symptom, and leave causes (CPU, memory, queue depth) on dashboards for diagnosis. The sharpest version is a burn-rate alert: page when you are spending the error budget fast enough to exhaust it early.
flowchart TB C(["CPU at 95"]) --> NO(["dashboard, not a page:
users may be fine"]) B(["burning budget:
2 pct in 1 hour"]) --> YES(["PAGE:
SLO will blow
if this holds"]) classDef cause fill:#FEF3C7,stroke:#1F2937,color:#0E1726; classDef ok fill:#D6F5E3,stroke:#1F2937,color:#0E1726; classDef bad fill:#FEE4E2,stroke:#1F2937,color:#0E1726; class C cause; class NO ok; class B,YES bad;
Don't wake the doctor because a machine beeped; wake them because the patient is in pain. Alert fatigue is what happens when every beep is a page, so make the page mean "a human is hurting right now".
"Burn rate" is how many times faster than budget you're failing. A rate of 1 exactly spends the month's budget in a month. Google's SRE pattern uses multiple windows: a short window confirms the burn is happening now, a long window confirms it's real and not a blip. High burn = page; low, sustained burn = ticket.
| Budget spent | Long window | Short window | Burn rate | Response |
|---|---|---|---|---|
| 2% in 1h | 1 hour | 5 min | 14.4× | page immediately |
| 5% in 6h | 6 hours | 30 min | 6× | page |
| 10% in 3d | 3 days | 6 hours | 1× | ticket, fix in hours |
A small leak that will empty the tank next month is a ticket. A gush that empties it by lunchtime is a phone call. Same fault, different urgency, and the burn rate is the number that tells them apart.
An incident is a process, not a scramble. Classify severity so the response matches the pain, assign clear roles so no one's both firefighting and updating the status page, and afterwards run a blameless postmortem that fixes the system, not the person.
flowchart LR DET(["Detect
alert fires"]) --> TRI(["Triage
SEV1/2/3, assign IC"]) TRI --> MIT(["Mitigate
stop the bleeding:
rollback, shed load"]) MIT --> RES(["Resolve
fix root cause"]) RES --> LRN(["Learn
blameless postmortem
+ action items"]) classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726; classDef step fill:#DCEBFE,stroke:#1F2937,color:#0E1726; classDef good fill:#D6F5E3,stroke:#1F2937,color:#0E1726; class DET io; class TRI,MIT,RES step; class LRN good;
Track how long each stage takes: time to detect, acknowledge, resolve (MTTD / MTTA / MTTR). Blameless means you ask "what let this happen and slip through", not "who did it", because people hide mistakes from blame and surface them for fixes.
Incident metrics are all "mean time to something", and the slippery one is MTTR: it can mean restore (service back, the number most teams track) or repair / resolve (root cause actually fixed). Pin the definition before you set a target, or two teams will report the same incident very differently.
| Metric | Stands for | Clock starts | Clock stops |
|---|---|---|---|
| MTTD | time to detect | failure begins | you know about it |
| MTTA | time to acknowledge | alert fires | a human owns it |
| MTTR | time to restore / repair | failure begins | users served again |
| MTBF | mean time between failures | service recovers | it breaks again |
Availability falls right out of these: MTBF ÷ (MTBF + MTTR) = uptime ÷ total time. Shrinking MTTD and MTTR buys you nines without ever making the system fail less often. Fast recovery beats rare failure, and it's usually cheaper to build.
Google's DORA program (the State of DevOps research, and the book Accelerate) found four metrics that predict software delivery performance, split across two axes: throughput (speed) and stability. The headline finding, and the one to quote in a leadership room: the best teams are fast and stable. Speed and safety are not a trade-off.
| Metric | Axis | Measures | Elite benchmark |
|---|---|---|---|
| Deployment frequency | throughput | how often you ship to prod | on-demand, many/day |
| Lead time for changes | throughput | commit to running in prod | under a day |
| Change failure rate | stability | % of deploys causing a failure | 0–5% |
| Failed-deploy recovery | stability | time to recover a bad deploy | under an hour |
Two speed metrics, two safety metrics. DORA later added a fifth, reliability (are you meeting your SLOs?), which ties this straight back to tonight. Adjacent frameworks worth a name: SPACE for developer productivity and Apdex for a single latency-satisfaction score.
Correct code is necessary, not sufficient. Operability is part of "done": before a service carries real traffic, it needs the means to be seen, alerted on, and recovered.
Take the checkout service from a booking app (the ClubCP case study). Walk the whole chain: pick the SLI, set the SLO, derive the budget, define the page, and point the runbook at the first place to look. This is exactly what the capstone asks for.
flowchart LR SLI(["SLI: successful
checkouts over attempts"]) --> SLO(["SLO: 99.9 over
28 days"]) SLO --> EB(["budget: 0.1 percent
~43 min per month"]) EB --> AL(["fast-burn page:
2 percent of budget
in 1 hour"]) AL --> RUN(["runbook: open the
checkout trace,
check payments span"]) classDef io fill:#FEF3C7,stroke:#1F2937,color:#0E1726; classDef step fill:#DCEBFE,stroke:#1F2937,color:#0E1726; classDef good fill:#D6F5E3,stroke:#1F2937,color:#0E1726; class SLI io; class SLO,EB,AL step; class RUN good;
One sentence becomes a working alarm: "count good checkouts, promise three nines a month, that buys 43 minutes of failure, and page me the moment I'm burning it fast enough to run out early, and here's the trace to open first." Reliability is now operational, not aspirational.
At scale, observability bills rival infra bills. The three cost drivers are log volume, metric cardinality, and trace retention. Control them deliberately, or the finance team controls them for you by turning things off.
Keeping every log line forever is like never emptying the bins because a receipt might matter someday. Decide what's worth keeping, for how long, and let the rest expire, at the Collector, before it hits the expensive backend.
Each one feels like observability while delivering none. Naming them in a review is the staff-level instinct: telemetry is only worth its cost if it changes a decision.
Pick the signals that track user pain (golden signals, RED, USE); turn them into SLIs and SLOs so reliability is a number; spend the error budget deliberately under a policy; use metrics, logs, traces, profiles and events for their strengths, instrumented once with OpenTelemetry; alert on symptoms and budget burn; run incidents blamelessly and measure recovery (MTTR); watch delivery health with DORA; and ship behind a readiness checklist.