Reliability · observability
S10 said it: reliability you can't measure is a wish. This sheet turns "it should be up" into numbers you can defend, alert on, and spend: SLIs, SLOs, SLAs and error budgets, the signals to watch (golden signals, RED, USE), the five pillars and OpenTelemetry, the tool landscape, multi-window burn-rate alerting, incident response and DORA. Companion to S10 · Observability & Operational Readiness.
Measure what the user feels, target it with a number, then spend the difference on purpose. An SLO turns reliability from an argument into arithmetic, and the error budget is the room it buys you to ship. Everything else, the signals, the pillars, the tools, the alerts, exists to defend that number and to investigate when it slips.
Monitoring is the dashboards you built for failures you predicted. Observability is being able to ask a question you never anticipated, from the data you already emit. Formally, it is how well you can infer a system's internal state from its outputs. Either way, reliability only becomes manageable once it is a number you agreed on in advance.
Pre-decided gauges for known failure modes: "is CPU over 80, is the queue backing up." Necessary, but it only answers questions you already thought to ask.
The ability to ask new questions of existing telemetry: "why are only Android users in one region seeing slow checkout?" You investigate without shipping new code.
Treating "we have dashboards" as observability. Dashboards answer the questions you anticipated. Real incidents are the ones you didn't, so the test is whether you can slice your telemetry by a dimension you didn't pre-build. And none of it matters until a number, the SLO, tells you whether things are actually fine.
Before SLIs, decide what is worth measuring at all. Three overlapping frameworks answer that. The golden signals are the idea; RED and USE are the recipes you apply per service and per resource.
| Framework | Origin | The signals | Apply to |
|---|---|---|---|
| Golden signals | Google SRE | latency, traffic, errors, saturation | any user-facing system: the first four graphs to build |
| RED | Tom Wilkie | Rate, Errors, Duration | every request-driven service; maps onto your SLIs |
| USE | Brendan Gregg | Utilisation, Saturation, Errors | every resource: CPU, memory, pool, disk, queue |
Golden signals are what to watch; RED tells you the service is hurting and how (alert on this, it is what the user feels); USE tells you which resource underneath is the cause (diagnose with this on a dashboard). Page on RED, investigate with USE.
Four linked ideas. Measure the signal (SLI), set your internal target (SLO), promise a looser one to customers (SLA), and the gap from your SLO to 100% is the budget you spend on risk and velocity.
| Term | Is | Audience | If missed |
|---|---|---|---|
| SLI | a ratio of good events to total, that tracks user pain | engineers | nothing, it is just the reading |
| SLO | your internal target on an SLI, over a window | the team | budget burns, freeze rule kicks in |
| SLA | a contractual promise to a customer | customers, legal | credits, refunds, penalties |
| Error budget | 1 minus the SLO; how much you may fail | the team | unspent means you move too slowly |
A worked SLO, written as one sentence
99.9% of checkout reads complete in under 200ms, over a rolling 28-day window. SLI = reads under 200ms ÷ all reads # the signal SLO = 99.9% target on that SLI # the internal promise SLA = 99.5% in the customer contract # looser, so the SLO warns first budget = 0.1% of reads may be slow ~ 43 min/mo # the room to take risk
SLI is the thermometer, SLO is the temperature you promised yourself to hold, SLA is the temperature in the contract with a fine attached, and the error budget is how far you may drift before you stop adding features and fix reliability. Keep the SLO stricter than the SLA so you are warned before you owe refunds.
Each nine you add to the SLO cuts the monthly allowance by ten. Knowing these by heart is the fastest way to sanity-check a target someone proposes in a review.
| SLO | Downtime / month | Downtime / year | Feels like |
|---|---|---|---|
| 99% | ~7h 18m | ~3.65 days | generous, internal tools |
| 99.9% | ~43m | ~8.8h | typical user-facing service |
| 99.95% | ~21m | ~4.4h | important revenue path |
| 99.99% | ~4m | ~52m | expensive, rarely needed |
| 99.999% | ~26s | ~5m | five nines, very few systems |
The budget is toothless without a written policy that product agreed to when heads were cool. It removes the argument from the moment of the incident.
The classic three (metrics, logs, traces) plus two now mainstream: continuous profiling and wide events. Each is best at one job. Reach for the wrong one and you either drown in cost or can't answer the question. Each card: answers the question it owns, use when to reach for it, watch the trap.
traceparent header); sample head or tail to control cost.Metric types: pick the right shape
| Type | Is | Example |
|---|---|---|
| Counter | monotonic total, query as a rate | http_requests_total |
| Gauge | a value now, up and down | queue_depth, mem_used |
| Histogram | bucketed, aggregate percentiles server-side | request_latency |
| Summary | client-side quantiles, can't merge across pods | legacy latency |
Percentiles > averages, and how to sample
Averages lie. A mean of 120ms can hide a p99 of 900ms. On a page with 100 assets, a 1-in-100 slow request is almost every page. SLO and alert on p99 / p99.9, never the mean, because you are your p99 to your busiest users.
Sampling traces: head sampling decides at the first span (cheap, but throws away rare errors); tail sampling buffers the whole trace and keeps it if it erred or was slow (needs a collector, catches the interesting ones).
The investigation path: metric to trace to log to profile
flowchart TB A["alert fires: p99 checkout up"] --> M["METRICS: what and how bad, the SLO is burning"] M --> T["TRACES: which hop is slow, payments span 540ms"] T --> L["LOGS: why that hop failed, timeout calling provider"] L --> P["PROFILE: why it is slow inside, a hot serialize loop"] P --> R["root cause"] classDef good fill:#D6F5E3,stroke:#1B7F46,color:#09244B class R good
OTel is the CNCF standard (a merge of OpenTracing and OpenCensus) for generating traces, metrics and logs. Instrument with vendor-neutral SDKs and auto-instrumentation, ship over the OTLP protocol to a Collector, and the Collector fans out to any backend. The payoff: your instrumentation no longer marries you to one vendor.
flowchart LR APP["your app: OTel SDK plus auto-instrument"] -->|OTLP| COL["OTel Collector: receive, batch, sample, redact"] COL --> PROM["Prometheus or Mimir: metrics"] COL --> TEMPO["Tempo or Jaeger: traces"] COL --> LOKI["Loki or ELK: logs"] COL --> SAAS["Datadog or Honeycomb: if you prefer SaaS"] classDef hub fill:#DCEBFE,stroke:#4A90D9,color:#09244B classDef out fill:#D6F5E3,stroke:#1B7F46,color:#09244B class COL hub class PROM,TEMPO,LOKI,SAAS out
OTel is a universal plug. Wire your app to the plug once; swap the wall socket (backend vendor) whenever you like without rewiring the house. The Collector is also where you control cost centrally: batch, tail-sample, drop high-cardinality labels, and redact PII before data hits an expensive backend.
You do not need all of it, one thing per job. The landscape splits into open-source components you run and SaaS you pay for, tied together by OpenTelemetry.
| Job | Open-source | SaaS / hosted |
|---|---|---|
| Instrument & collect | OpenTelemetry, Fluent Bit, Vector, eBPF (Pixie) | vendor agents (Datadog Agent) |
| Metrics | Prometheus, VictoriaMetrics, Mimir / Thanos / Cortex | Datadog, Chronosphere |
| Dashboards | Grafana | Datadog, New Relic, Kibana |
| Logs | Loki, Elasticsearch / OpenSearch | Splunk, Datadog, Sumo Logic |
| Traces | Jaeger, Tempo, Zipkin | Datadog APM, Honeycomb, Lightstep |
| Profiling | Pyroscope, Parca | Datadog, Polar Signals |
| Errors | GlitchTip | Sentry, Rollbar, Bugsnag |
| Alerting & on-call | Alertmanager, Grafana OnCall | PagerDuty, Opsgenie |
| Stack | What | Wins when | Watch out |
|---|---|---|---|
| OSS "LGTM" | Prometheus + Grafana + Loki + Tempo + Alertmanager, fed by OTel | cost-sensitive, own your data, k8s-native | you run it: storage, upgrades, on-call for the tooling |
| SaaS all-in-one | Datadog, New Relic, Honeycomb | small team, want it working today, correlation built-in | cost scales with hosts + cardinality; bill shock |
| Cloud-native | CloudWatch, Google Cloud Ops, Azure Monitor | single-cloud, basics only, zero extra vendors | weaker tracing/UX, poor cross-cloud, lock-in |
Self-host (OSS) to save money and own the data if you have hands to run it. Pay a SaaS to skip the plumbing if you have budget not time. Use the cloud's built-in if you are small and single-cloud. All three are defensible; hand-rolling twelve disconnected tools is not.
High CPU is not an incident; a user who can't check out is. Page on the symptom and on budget burn; leave causes on dashboards. Every page should mean "a human is hurting right now," or people learn to ignore the pager.
flowchart TB
subgraph PAGE["page on a symptom or burn"]
P1["SLO error budget burning fast"]
P2["checkout success rate dropping"]
P3["p99 latency over the SLO"]
end
subgraph DASH["keep on a dashboard"]
D1["CPU at 95 percent"]
D2["memory at 70 percent"]
D3["queue depth rising slowly"]
D4["cache hit ratio"]
end
RT["rule of thumb: if users do not feel it yet, it is not a page"]
P3 ~~~ D1
D4 ~~~ RT
classDef bad fill:#FEE4E2,stroke:#C8102E,color:#09244B
classDef accent fill:#DCEBFE,stroke:#4A90D9,color:#09244B
class P1,P2,P3 bad
class RT accent
Multi-window burn-rate alerts (the SRE standard)
| Budget spent | Long window | Short window | Burn rate | Response |
|---|---|---|---|---|
| 2% in 1h | 1 hour | 5 min | 14.4× | page immediately |
| 5% in 6h | 6 hours | 30 min | 6× | page |
| 10% in 3d | 3 days | 6 hours | 1× | ticket, fix in hours |
Burn rate is how many times faster than budget you are failing; a rate of 1 spends the whole window's budget exactly on time. The short window confirms it is happening now, the long window confirms it is real and not a blip. Fewer, sharper, symptom-based alerts beat a wall of cause-based ones. If an alert never needs human action, it should be a dashboard or a ticket, not a page.
An incident is a process, not a scramble. Classify severity, assign clear roles, mitigate before you root-cause, and learn blamelessly. Then measure how you did with the MTTx metrics.
flowchart LR DET["Detect: alert fires"] --> TRI["Triage: set SEV, assign an incident commander"] TRI --> MIT["Mitigate: stop the bleeding, rollback or shed load"] MIT --> RES["Resolve: fix the root cause"] RES --> LRN["Learn: blameless postmortem, action items"] classDef good fill:#D6F5E3,stroke:#1B7F46,color:#09244B class LRN good
The MTTx family: know which clock you mean
| Metric | Clock starts | Clock stops |
|---|---|---|
| MTTD detect | failure begins | you know about it |
| MTTA acknowledge | alert fires | a human owns it |
| MTTR restore | failure begins | users served again |
| MTBF between | service recovers | it breaks again |
MTTR is ambiguous, and availability falls out of it
MTTR can mean restore (service back, what most teams track) or repair / resolve (root cause fixed). Pin the definition before you set a target, or two teams report the same incident very differently.
availability = MTBF ÷ (MTBF + MTTR)
= uptime ÷ total time
Shrinking MTTD and MTTR buys nines without making the system fail less often. Fast recovery beats rare failure, and it is usually cheaper to build.
Ask "what let this happen and slip through," not "who did it." People hide mistakes from blame and surface them for fixes. Severity (SEV1/2/3) sizes the response; the incident commander coordinates so no one is both firefighting and updating the status page.
Zoom out from one service to the whole delivery system. Google's DORA research (the State of DevOps reports, and the book Accelerate) found four metrics that predict performance, split across two axes: throughput (speed) and stability. The headline: elite teams are fast and stable. Speed and safety are not a trade-off.
| Metric | Axis | Measures | Elite benchmark |
|---|---|---|---|
| Deployment frequency | throughput | how often you ship to prod | on-demand, many/day |
| Lead time for changes | throughput | commit to running in prod | under a day |
| Change failure rate | stability | % of deploys causing a failure | 0–5% |
| Failed-deploy recovery | stability | time to recover a bad deploy | under an hour |
DORA later added reliability (are you meeting your SLOs?) as a fifth key, connecting delivery performance straight to this sheet. Adjacent frameworks worth a name: SPACE for developer productivity (satisfaction, performance, activity, communication, efficiency) and Apdex for a single latency-satisfaction score. Use DORA to argue that investing in observability and CI/CD makes teams both faster and safer.
Correct code is necessary, not sufficient. Before a service carries real traffic it needs the means to be seen, alerted on, and recovered. Steal this list for the capstone.
Before launch, you have
The mindset
"Done" includes operability. A feature that works in the happy path but can't be observed, alerted on, or rolled back is not ready, it is a future incident with no instrumentation.
The readiness review is where a senior engineer earns trust: not "does it work" but "when it breaks at 3am, can the on-call see it, understand it, and recover, without you?"
At scale, observability bills rival infra bills. Control the three cost drivers deliberately, at the Collector, before data hits the expensive backend, or finance controls them for you by turning things off.
Cardinality
Drop high-variety labels. The user-id tag is the classic five-figure bill-killer. Keep metric labels low-cardinality; push high-cardinality detail into wide events or logs.
Sampling
Tail-sample traces: keep errors and slow requests, drop the boring 99%. Sample chatty logs. You rarely need every successful request stored forever.
Retention tiers
Hot storage for 7 days, cold or archive after; aggregate old metrics down to lower resolution. Decide what is worth keeping, and for how long.
Start from the question you are actually asking and reach for the telemetry that answers it.
flowchart TB
Q{"what are you trying to answer?"}
Q --> A1["is the service healthy right now?"] --> R1["METRICS RED plus the SLO"]
Q --> A2["how much budget have we burned?"] --> R2["error budget plus burn rate"]
Q --> A3["which hop made this request slow?"] --> R3["TRACES, spans"]
Q --> A4["why did that specific call fail?"] --> R4["LOGS, structured by field"]
Q --> A5["why is one service slow with no external cause?"] --> R5["PROFILE, flame graph"]
Q --> A6["which resource is the bottleneck?"] --> R6["METRICS USE"]
Q --> A7["what did the slow requests have in common?"] --> R7["WIDE EVENTS, slice by field"]
classDef accent fill:#DCEBFE,stroke:#4A90D9,color:#09244B
class Q accent
| You want to… | Reach for | Not |
|---|---|---|
| Know if users are in pain | SLI / SLO (RED) | raw CPU graphs |
| Decide whether to freeze features | Error budget + policy | gut feel |
| Find the slow service in a request | Traces | grepping logs blindly |
| Explain one failure in detail | Structured logs | a metric counter |
| Speed up one hot service | Continuous profiling | a distributed trace |
| Diagnose a saturated resource | USE metrics | a trace |
| Slice an incident by any dimension | Wide events | pre-aggregated metrics |
| Decide what to page on | Symptom / burn rate | cause thresholds |
The observability questions that come up, and the sentence or two that shows you understand the trade-off rather than reciting a term.
Q · SLI vs SLO vs SLA vs error budget?
SLI is the measured signal (good ÷ total). SLO is your internal target on it. SLA is the looser customer contract with penalties. The error budget is 1 minus the SLO. Keep the SLO stricter than the SLA so you are warned before refunds.
Q · Golden signals vs RED vs USE?
Golden signals (latency, traffic, errors, saturation) are the idea. RED (rate, errors, duration) applies it per service; USE (utilisation, saturation, errors) applies it per resource. Alert on RED, diagnose with USE.
Q · Why is 100% the wrong SLO?
It leaves zero error budget, so it forbids all risk, all deploys, all velocity, and you still can't hit it because dependencies fail. Pick the lowest number users won't notice.
Q · Metrics, logs, traces or profiles, for what?
Metrics for what and how bad (cheap, SLOs). Traces for where the time went across services. Logs for why one call failed. Profiles for why one service is slow inside. Walk them in that order.
Q · What is OpenTelemetry?
The CNCF standard for emitting traces, metrics and logs. You instrument once with vendor-neutral SDKs, ship OTLP to a Collector, and fan out to any backend, so instrumentation no longer locks you to one vendor.
Q · Head vs tail sampling?
Head decides at the first span (cheap, but discards rare errors before you know they mattered). Tail buffers the whole trace and keeps it if it erred or was slow (needs a collector, catches the interesting ones).
Q · What is a burn-rate alert?
An alert on how fast you are spending the error budget, not a static threshold. Multi-window (short confirms now, long confirms real): fast burn pages, slow burn tickets. It catches both the spike and the slow leak.
Q · Why not alert on CPU?
CPU is a cause, not a symptom; high CPU with happy users is fine. Page on what users feel (the SLO burning), keep CPU on a dashboard for diagnosis. It is how you avoid alert fatigue.
Q · What does MTTR actually mean?
It is ambiguous: restore (service back) or repair (root cause fixed). Say which. Availability = MTBF ÷ (MTBF + MTTR), so cutting detection and recovery time buys nines without reducing how often you break.
Q · What are the DORA metrics?
Four delivery metrics on two axes: deployment frequency and lead time (speed), change failure rate and recovery time (stability). The finding: elite teams are fast and stable. A fifth, reliability, ties to your SLOs.
Q · What breaks a metrics system?
Cardinality. Tagging a metric with a high-variety label (user id, request id) multiplies it into millions of series, blowing up cost and query time. Keep labels low-cardinality; use wide events for the detail.
Q · Counter vs histogram?
A counter only climbs; you read it as a rate. A histogram buckets observations so you can aggregate percentiles across instances, which a client-side summary can't. Use histograms for latency SLIs.
The ways observability goes wrong, and the habit that catches all of them.
The traps
The repeatable move
1. pick the signals users feel (golden / RED)
2. set an SLI + SLO + error budget on them
3. instrument the pillars once via OpenTelemetry
4. alert on the symptom / burn, not the cause
5. run incidents blamelessly, measure MTTR
The tell in a design review: an alert on a cause (CPU, memory) with no SLO behind it, or a metric tagged with something high-cardinality. Spot it and push back.
Measure what users feel, target it, spend the budget on purpose, and page only on pain. This sheet is the S10 "reliability you can't measure is a wish" slide unpacked, and resilience (S9) is what you do when the budget starts burning.
Companion to S10 · Observability & Operational Readiness. Further reading: Google SRE · Service Level Objectives · SRE Workbook · Alerting on SLOs · The RED method · Brendan Gregg · The USE method · OpenTelemetry · Observability primer · DORA · software delivery metrics.