Module 13 · Lesson 3 of 15Free

SLOs and error budgets: defining reliability targets

Level

Intermediate

Topic

Observability, incidents and safe deployments

Reading time

18 min

Kartly reports 99.95% successful API requests during a festival sale. Yet checkout is failing for one customer in ten. Product-page traffic is hiding the failures that stop people buying.

Should the team continue releasing features, or stop and repair checkout? This lesson answers one question: how do you turn customer expectations into a measurable reliability target and a rule for acting on failures?

The scenario uses Kartly’s many-services stage. All targets and measurements below are illustrative, not production statistics.

Count the outcome customers need

A site-wide success percentage treats every request as equally important. That is convenient to calculate, but it answers the wrong question here.

Suppose successful catalog reads dominate Kartly’s traffic. Adding more catalog traffic improves the reported percentage without fixing a single checkout.

A service level indicator, or SLI, is the measurement. For this lesson, it is the fraction of eligible checkout requests with a good outcome.

A service level objective, or SLO, sets the target for that measurement over a defined window. Expressing an SLI as good events divided by total events makes its relationship to an error budget explicit.^{1Implementing SLOsSteven Thurgood and David Ferguson, The Site Reliability Workbook, 2018}

You already selected useful service signals in RED and USE methods. Now you must decide which signal represents an experience worth protecting.

Start with one customer task, not every metric on the dashboard. CPU utilization helps explain failures; it does not tell you whether someone could place an order.

The next difficulty is deciding what “good” actually means.

Define good before choosing a percentage

Kartly’s first proposal is “checkout returns HTTP 200.” That rule would count a fast response containing an application error as success.

Instead, the team defines a good response as an expected checkout result delivered within two seconds. An expected result includes a confirmed order or an explicit business rejection, such as insufficient stock.

A payment timeout is not an expected rejection. The customer still does not know whether the purchase worked.

This boundary also has limits. It measures the checkout API, not the whole phone experience. A slow screen or a failed connection before Kartly’s edge remains outside this measurement.

Server-side observations can miss requests that never reach the server. Client measurements and external probes expose different parts of that gap.^{1Implementing SLOsSteven Thurgood and David Ferguson, The Site Reliability Workbook, 2018}

Write the measurement contract before implementing the counters:

Eligibility must not depend on whether the service succeeded. Otherwise, a failure could remove itself from the denominator.

An eligible request rejected during overload counts as bad. Planned maintenance also counts under this contract: customers still cannot buy.

This contract makes the percentage meaningful. It does not yet explain whether the proposed target is worth its cost.

A target creates a failure allowance

For a good-event ratio, the error budget is the failure allowance implied by the target. It is not a separate reliability target.^{1Implementing SLOsSteven Thurgood and David Ferguson, The Site Reliability Workbook, 2018}

Kartly proposes 99.9% good checkout API outcomes. That leaves 0.1% for outcomes outside the contract.

For one illustrative trailing window:

A request-based budget changes as the window’s total traffic changes. It is not a fixed allocation issued at the start of a month.^{2Service level objectives (SLOs)Amazon Web Services, Amazon CloudWatch documentation}

Moving from 99.9% to 99.99% would reduce this window’s allowance to 3,000 bad requests. Kartly’s observed failures would exceed it substantially.

That tighter target might justify expensive improvements. It does not create them.

Keep correctness requirements separate. A duplicate charge is not an acceptable purchase from a general failure allowance. Investigate it even when the API SLO is met.

The budget guides risk-taking. It does not authorize every kind of failure.

Requests are not minutes

“99.9% means about 43 minutes of downtime a month” is only appropriate for a matching time-based definition and month length.

For an illustrative 30-day time-based objective, the arithmetic is:

A quiet-hour outage and a sale-hour outage can last equally long but affect different numbers of purchases.

A request-based SLO counts good requests divided by total requests. A period-based SLO counts good periods divided by total periods, using a rule that decides whether each period was good.^{2Service level objectives (SLOs)Amazon Web Services, Amazon CloudWatch documentation}

Neither is universally correct. A continuously available connection may need a time-oriented measure. Kartly chooses requests because each failed checkout submission represents a purchase interrupted.

Do not accidentally create a third calculation by averaging minute-level success percentages.

Suppose one minute contains 100 requests, all good. Another contains 9,900 requests, with 99 bad.

The combined result is 9,901 good requests out of 10,000: 99.01%. Averaging the minute percentages gives 99.5%, because it gives the quiet minute equal weight.

Sum good counts and total counts across instances and time. Divide only after aggregation.

The same rule applies across services. A healthy catalog must not outweigh broken checkout merely because it serves more requests.

Even correctly weighted counts need a trustworthy collection path.

Make the measurement reproducible

Kartly implements the contract with fixed-size time buckets. Each bucket stores eligible and good counts for a specific checkout-contract version.

The collection path builds on logs, metrics and traces; this lesson only specifies what the SLO calculation needs.

1. The edge decides eligibility when accepting a checkout request.
2. It starts a timer and waits for a classified result.
3. At completion or timeout, it records exactly one terminal observation.
4. The aggregation process sums the window’s eligible and good counts.
5. The evaluator calculates the ratio, allowance and remaining budget.

A late response must not create a second observation after a timeout. For this design, an edge-local completion guard lets only the first terminal event record the request.

The calculation is compact enough to inspect directly:

{
  "contract_version": 1,
  "window_days": 30,
  "target_good_fraction": 0.999,
  "eligible_requests": 30000000,
  "good_requests": 29976000,
  "bad_requests": 24000,
  "allowed_bad_requests": 30000,
  "remaining_bad_requests": 6000
}

These are illustrative evaluated values, not a vendor configuration format.

For latency, a histogram bucket at the exact deadline can count observations within that threshold. An interpolated percentile is unnecessary for this calculation.^{3Histograms and summariesPrometheus authors, Prometheus documentation}

Kartly’s good count must also require an expected outcome. A fast internal error remains bad. Separate latency and success totals cannot tell you how many requests satisfied both conditions.

Kartly does not pause checkout because its reporting system fails. It pauses automated release approval and restores measurement coverage.

A percentage without coverage information can encourage the wrong release decision.

A rolling window keeps moving

Kartly’s rolling window always covers the previous 30 days. Every evaluation adds recent buckets and removes buckets that have become too old.

This means a budget can recover when old failures leave the window. It can also change when traffic rises or falls.

For comparison, a calendar-month window gives a familiar reporting boundary. But an incident near the boundary can disappear from the new month’s report immediately.

Kartly chooses a rolling window for release decisions to avoid that abrupt reset. Calendar reports can still serve business reporting.

A recovered budget does not prove that the underlying bug was fixed. The team tracks corrective work separately from the arithmetic.

Retries introduce another boundary. Kartly counts each eligible HTTP attempt, including retries using the same idempotency key.

One failed attempt followed by one successful retry produces a 50% good-event ratio for those attempts. A customer-task measurement might record eventual success instead.

Neither result is mathematically wrong. They answer different questions. Kartly retains the attempt-based API objective and avoids calling it a unique-customer success rate.

The remaining question is what the team does when failures consume the allowance quickly.

Turn consumption into engineering action

Kartly agrees on a written policy before the next incident. An error-budget policy connects measured performance to explicit actions and ownership.^{1Implementing SLOsSteven Thurgood and David Ferguson, The Site Reliability Workbook, 2018}

For this scenario:

A freeze that blocks the repair would defeat its purpose.

Consider the window with 6,000 bad requests remaining. An incident adds 15,000 bad requests. Assume other window counts stay unchanged except for those incident requests.

The budget becomes negative, even after accounting for the small allowance added by those new requests. Kartly has breached its target.

The team should not wait for that point to notice trouble.

Burn rate compares the observed bad-event fraction with the fraction allowed by the target. A value above one means failures are occurring faster than the target permits proportionally.^{2Service level objectives (SLOs)Amazon Web Services, Amazon CloudWatch documentation}

For this lesson’s target, an observed bad-event fraction of 2% gives a burn rate of 20.

That number does not predict an exact exhaustion time. Request volume changes, and old observations leave the rolling window.

It does tell you that the recent failure fraction is far above the sustainable fraction. Alerting: signals worth paging for develops the windows and thresholds needed to act on that signal.

Where this shows up

• Design: monitoring and SLOs for a service turns the measurement contract into dashboards and operational rules.
• Design: an e-commerce checkout uses customer outcomes to distinguish a functioning purchase flow from healthy individual components.

Kartly now has a reliability target; next it needs alerts that warn about threatened budgets without paging for every fluctuation.

Common mistakes

• Choosing the percentage first. Define eligible and good events before discussing how many failures are acceptable.
• Treating the budget as permission to ignore customers. Concentrated harm can demand action even when the overall target is met.
• Changing exclusions after an incident. Version the contract prospectively instead of making the failed requests disappear.
• Reporting only the aggregate. Inspect important regions and client versions, using the same definitions, to find concentrated failures.
• Treating an SLO as a correctness guarantee. A high good-event ratio does not prove that orders or charges are correct.

These mistakes usually produce reassuring numbers rather than useful decisions.

Check your understanding

Summary