Module 01 · Lesson 10 of 13Free

Model API errors, rate limits and retries

Level

Intermediate

Topic

AI engineering fundamentals

Reading time

17 min

Sale day: 9,000 summaries and a wall of 429s

Deskora is a help desk app. A shop’s support team uses it to answer customer emails from one shared inbox. Each customer problem becomes a ticket: one customer’s whole conversation with the support team, message by message. When a ticket gets long, a background worker asks a model for a three-line summary, so the agent who picks it up can reply sooner.

At 10:00 one morning, hundreds of Deskora’s shops start a seasonal sale at the same time. Priya’s kitchenware shop is one of them. For six minutes, about 1,500 tickets a minute grow long enough to need a summary.

Tomás, the Deskora engineer who built the summaries, sees the error log fill with 429 rate_limit_error. Hidden among them are 404 not_found_error lines: a config change sent some jobs the model id claude-haiku-5.5 instead of claude-haiku-5-5.

At 10:40, Priya opens a 41-message ticket from a customer, Amara. The summary box at the top is empty. She reads all 41 messages anyway.

This lesson answers one question: which failures should Deskora retry, how should it retry them, and what does Priya see when the retries run out?

Deskora’s 120,000 tickets a day is a fact used across this course. The ticket sizes, the sale-day numbers, Deskora’s usage tier and the timings below are estimates for this lesson’s scenario.

Two kinds of failure: wait it out or fix it

The model API answers a failed call with an HTTP status code and a JSON body. The body’s error.type names the problem, and request_id identifies the call for support ^{1ErrorsAnthropic, Claude API documentation}:

{
  "type": "error",
  "error": {"type": "rate_limit_error", "message": "..."},
  "request_id": "req_011CSHoEeqs5C35K2UUqR7Fy"
}

Ask one question of each error: would the same request, sent again later, succeed? If yes, it is a retryable error, caused by load or a passing fault. If no, the request or the account is wrong.

These are the errors the Claude API documents, as of October 2026 ^{1ErrorsAnthropic, Claude API documentation}:

One row needs care. A 429 usually means “slow down”, but the same status also arrives when Deskora’s monthly spend cap is reached. That 429 has no retry-after header, carries error.details.error_code: "enforced_spend_limit_reached", and keeps failing until the next month ^{2Rate limitsAnthropic, Claude API documentation}. A spend limit Deskora sets itself returns a 400 instead.

Retrying a fixable error is not harmless: the claude-haiku-5.5 jobs fail on every attempt, and each retry hides the bug longer.

To retry a 429 well, you need to know what is counted.

What the rate limit counts

Rate limiting caps how much work an organization may send. On the Claude Messages API, each model class has three limits: requests per minute (RPM), input tokens per minute (ITPM) and output tokens per minute (OTPM) ^{2Rate limitsAnthropic, Claude API documentation}. Exceed any one, and the call gets a 429 that says which limit it hit.

The limits work like a token bucket. Capacity refills continuously, up to the maximum, rather than resetting at the top of each minute. A limit of 60 requests a minute may also be enforced as one a second, so a short burst can fail even when the minute’s total is fine ^{2Rate limitsAnthropic, Claude API documentation}. The bucket mechanics are taught in Rate limiting algorithms.

For most current models, only uncached input tokens count toward ITPM, and only tokens actually produced count toward OTPM ^{2Rate limitsAnthropic, Claude API documentation}.

Assume Deskora is on the Build tier. As of October 2026, its limits for Claude Haiku 5.5 are 5,000 RPM, 5,000,000 ITPM and 1,000,000 OTPM ^{2Rate limitsAnthropic, Claude API documentation}.

Input tokens run out first, while the request count sits at 30%. The provider will accept 625 summaries a minute however the calls are sent. Retries can’t raise that number; they only decide which jobs get the space, and when.

A sudden jump can also cause 429s: the API applies acceleration limits when usage rises sharply, and the docs advise ramping up gradually ^{1ErrorsAnthropic, Claude API documentation}. A sale that starts at 10:00 sharp is that kind of jump.

A 429 from a rate limit usually carries a retry-after header: the number of seconds to wait, since earlier retries will fail ^{2Rate limitsAnthropic, Claude API documentation}. Every response also carries anthropic-ratelimit-* headers showing what remains. Slowing down by reading those headers is pacing, covered with queues in Provider rate limits, queues and fallbacks.

Waiting works, if the wait is long enough. Tomás’s first version didn’t wait long enough.

Retrying on top of retrying

A common belief is that more retries make a job more reliable. Here they can’t. The bucket accepts 625 summaries a minute whether each job tries 3 times or 15. Extra attempts only add load and noise. What was missing was time: retries spread over the 15 minutes the backlog lasts, not over one minute.

Waiting longer between tries is the job of exponential backoff: each retry waits up to twice as long as the one before, up to a cap. Retry jitter picks a random delay inside that window, so the 875 jobs that failed together don’t all come back in the same second ^{5Exponential Backoff And JitterMarc Brooker, AWS Architecture Blog, 2015}. Full jitter, caps and retry amplification are taught in Exponential backoff and jitter.

Where the retries live

The SDK’s built-in retries cover connection errors, 408, 409, 429 and every status of 500 or above, which includes 529. They use a short exponential backoff and honour retry-after ^{3Python SDKAnthropic, Claude API documentation}. You set two numbers on the client: max_retries (default 2) and timeout (default 10 minutes). Timed-out requests are retried too ^{3Python SDKAnthropic, Claude API documentation}.

Here is the job function. The job queue calls it, and reschedules the job when it returns "later". It never stores an empty or cut-off summary.

import random
import anthropic

# Seconds-scale retries stay in the SDK. 60 s is far above a normal summary's time.
client = anthropic.Anthropic(max_retries=2, timeout=60.0)

SYSTEM = "Summarize the support ticket in exactly three lines."
MAX_JOB_ATTEMPTS = 5
TRY_LATER = (anthropic.InternalServerError,  # 500, 503, 504 and other 5xx
             anthropic.OverloadedError,      # 529
             anthropic.APIConnectionError)   # includes APITimeoutError


def spend_cap_reached(err: anthropic.RateLimitError) -> bool:
    body = err.body if isinstance(err.body, dict) else {}
    details = body.get("error", {}).get("details") or {}
    return details.get("error_code") == "enforced_spend_limit_reached"


def later(attempt: int) -> tuple[str, str | float]:
    if attempt >= MAX_JOB_ATTEMPTS:
        return ("failed", "gave up: leave the ticket without a summary")
    window = min(30.0, 2.0 * 2 ** (attempt - 1))  # 2, 4, 8, 16 minutes, cap 30
    return ("later", random.uniform(0, window))  # full jitter


def run_summary_job(thread: str, attempt: int) -> tuple[str, str | float]:
    """Returns ("done", summary), ("later", minutes) or ("failed", reason)."""
    try:
        response = client.messages.create(
            model="claude-haiku-5-5", max_tokens=2000, system=SYSTEM,
            messages=[{"role": "user", "content": thread}],
        )
    except anthropic.RateLimitError as err:
        if spend_cap_reached(err):
            return ("failed", "spend cap reached: pause all summary jobs")
        return later(attempt)
    except TRY_LATER:
        return later(attempt)
    except anthropic.APIStatusError as err:  # 400, 401, 403, 404, 413
        return ("failed", f"{err.status_code}: fix and alert, don't retry")
    if response.stop_reason != "end_turn":
        return ("failed", f"stop_reason={response.stop_reason}")
    return ("done", "".join(b.text for b in response.content if b.type == "text"))

The order of the except clauses matters. RateLimitError and InternalServerError are both kinds of APIStatusError, so they must come before it. The last check is the one from Calling a model API: a 200 with a stop_reason other than end_turn is not a finished summary.

Every retry that runs is billed

A retry is a new request. If it reaches the model, it is billed like any other call. A 429 is rejected before the model runs, so it produces no tokens. A call your client gives up on is different: the provider may finish the work and bill it, even though nobody reads the answer.

That makes the timeout a cost setting. Suppose Tomás had set timeout=10.0, and a long ticket’s summary takes about 12 seconds (an assumption for this example).

So set the timeout well above the slowest normal call: 60 seconds here. A completed summary costs the price per call times the billed calls it took; the next lesson builds that into the estimate.

When retries run out, show no summary

After 5 attempts, about 30 minutes in the worst case, the job stops. What should Priya see then?

Deskora shows no summary. A missing summary costs Priya five minutes of reading. A wrong one can cost Amara a wrong reply. Calling a second provider when retries run out is a fallback, covered in the later lesson on queues and fallbacks linked above. A model that declines to answer is a different failure, covered in Refusals and fallbacks.

Each summary job moves through these states:

A failed job never writes to the summary field. A fixable error also pages an engineer, because no amount of waiting fixes it. The wrong model id would have paged Tomás at 10:01.

Try it yourself

Send a call with the wrong model id and read the error. A rejected request produces no tokens. If you don’t have an API key, skip this.

curl -sS -i https://api.anthropic.com/v1/messages \
  -H "content-type: application/json" \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2026-10-21" \
  -d '{"model": "claude-haiku-5.5", "max_tokens": 200, "messages": [{"role": "user", "content": "Say hello."}]}'

One run returned this, shortened; your request id and message will differ:

HTTP/2 404
request-id: req_011C...
content-type: application/json

{"type":"error","error":{"type":"not_found_error","message":"model: claude-haiku-5.5"},"request_id":"req_011C..."}

What to look for:

• The status and error.type: a 404 that no retry will fix.
• Change the model to claude-haiku-5-5 and run it again. It costs at most about $0.0001: 200 output tokens × $0.50 per million, plus a few input tokens.
• In that response, find anthropic-ratelimit-input-tokens-remaining and anthropic-ratelimit-requests-remaining. They show how much of the bucket is left.

Where this shows up

• Design: a ticket summary feature puts these job states into the full feature, with the queue and the storage.
• Design: an AI customer support platform makes several calls per ticket, so one rate limit is shared by many features.
• Design: scaling a chat assistant to a million users meets rate limits with a person waiting on every call.

Retries, timeouts and thinking all change what a summary really costs and how long it takes. The next lesson, Estimating the cost and latency of a feature, puts them into one estimate before launch.

Common mistakes

• Retrying every exception. A 400, 401 or 404 fails on every attempt. Fail at once, and alert a person.
• Wrapping the SDK in another retry loop. The SDK already retries twice. Set max_retries and timeout on the client instead.
• Treating every 429 the same. The spend-cap 429 has no retry-after and carries enforced_spend_limit_reached. It lasts until the next month.
• Saving whatever came back. An empty string, an error message or a cut-off summary must never reach the summary field.

Check your understanding

Summary