Module 01 · Lesson 11 of 13Free

Estimating the cost and latency of a feature

Level

Beginner

Topic

AI engineering fundamentals

Reading time

15 min

The launch question has two numbers

Deskora is a help desk app. A shop’s support team uses it to answer customer emails from one shared inbox. Each customer problem becomes a ticket: one customer’s whole conversation with the support team, message by message.

Tomás, a Deskora engineer, has built a short summary at the top of long tickets. Support agents like Priya read it instead of the whole thread.

Launch is in two weeks. Deskora’s product lead asks Tomás: what will summaries cost each month, and how fast will they be?

From his first test in Calling a model API, Tomás knows one summary costs about $0.0006 on Claude Haiku 5.5: about $38 a day for 40,000 summaries. He almost replies “about $1,140 a month.”

That number prices one call, not a feature that runs all day on tickets that keep growing. This lesson answers one question: how do you turn the cost and time of one call into a monthly budget and a latency promise you can keep?

Of Deskora’s 120,000 new tickets a day, 40,000 get a summary. That share and every other number below are estimates or assumptions for this lesson’s scenario.

One summary is many calls

The unit cost to estimate is one completed task, not one API call. Here the task is “Priya opens a long ticket and finds a summary that matches it.” Several calls stand behind it.

• The first summary. Deskora makes it in the background when a ticket reaches 8 messages, the threshold the design in the next lesson uses.
• Refreshes. A summary of messages 1–8 says nothing about message 30, so Deskora makes a new one as the ticket grows.
• Retries. Calls that fail with a 429 or a 5xx are sent again; the Python SDK retries them twice by default ^{5anthropic-sdk-pythonAnthropic, GitHub}. See Model API errors, rate limits and retries.

Each refresh is a new one-turn call with the whole thread, not a follow-up question. So earlier thinking is not resent, but the thread is, and it is longer every time.

How often to refresh is Tomás’s choice, and it moves the bill more than anything else.

The refresh rule sets the bill

Typical and long tickets, priced

Tomás prices two cases, using the shared token numbers from lesson 03. As of October 2026, Claude Haiku 5.5 costs $0.10 per million input tokens and $0.50 per million output tokens, for prompts up to 100,000 tokens ^{1PricingAnthropic, Claude API documentation}. Thinking tokens are billed as output tokens ^{4Steering thinkingAnthropic, Claude API documentation}.

The thinking amounts are the weakest assumption. If thinking runs at twice the assumed amount, the month is about $7,700. After launch, usage.output_tokens_details.thinking_tokens reports the real amount on every call ^{4Steering thinkingAnthropic, Claude API documentation}.

A long ticket is 3 times longer than a typical one but costs about 10 times as much. It gets more calls, and each call sends more, so cost grows with roughly the square of length. One average ticket would sit where little of the money is.

Tomás keeps the estimate as code, so he can rerun it with logged numbers after launch:

from dataclasses import dataclass

# claude-haiku-5-5, US dollars per million tokens, as of October 2026.
# Valid for prompts up to 100,000 tokens; above that Haiku 5.5 charges 5x.
INPUT_PRICE = 0.10
OUTPUT_PRICE = 0.50  # thinking tokens are billed at this price too

SYSTEM_TOKENS = 150
FIRST_AT = 8         # first summary at message 8
REFRESH_EVERY = 5    # new summary after every 5 new messages
RETRY_RATE = 0.02    # share of calls sent again, each billed in full
SUMMARIZED_PER_DAY = 40_000


@dataclass
class TicketCase:
    name: str
    share: float             # fraction of summarized tickets
    messages: int            # messages when the ticket goes quiet
    tokens_per_message: int
    thinking_tokens: int     # assumed, per call
    summary_tokens: int      # assumed, per call


def cost_per_ticket(case: TicketCase) -> float:
    total = 0.0
    for n in range(FIRST_AT, case.messages + 1, REFRESH_EVERY):
        input_tokens = SYSTEM_TOKENS + n * case.tokens_per_message
        output_tokens = case.thinking_tokens + case.summary_tokens
        total += (input_tokens * INPUT_PRICE + output_tokens * OUTPUT_PRICE) / 1_000_000
    return total * (1 + RETRY_RATE)


cases = [
    TicketCase("typical", 0.75, 20, 200, 300, 80),
    TicketCase("long", 0.25, 60, 250, 800, 100),
]
daily = 0.0
for case in cases:
    per_ticket = cost_per_ticket(case)
    day = per_ticket * case.share * SUMMARIZED_PER_DAY
    daily += day
    print(f"{case.name}: ${per_ticket:.5f} per ticket, ${day:,.2f} a day")
print(f"total: ${daily:,.2f} a day, about ${daily * 30:,.0f} a month")

The budget is ready. The second question is time.

Thinking is most of the wait

A summary’s latency has four parts. The job waits in Deskora’s queue for a free background worker. The model reads the prompt; time to first token is the time from sending the request to the first output token ^{3Reducing latencyAnthropic, Claude API documentation}. Then it writes its thinking, and only then the summary.

Writing time is output tokens divided by output speed, in tokens per second. Latency depends on prompt length, output length and thinking effort ^{2Models overviewAnthropic, Claude API documentation}. The provider ranks models by speed but gives no seconds, so every time below is an assumption to measure.

On the long ticket, thinking is 8 of the 12 seconds. A shorter prompt barely moves this budget; less thinking moves it most.

If Priya waited for the summary, streaming would help little: thinking comes before any text, so she would see an empty box for about 9 seconds.

Promise freshness, not speed

Deskora doesn’t make her wait. The summary is made in the background, before anyone opens the ticket, so 12 seconds versus 6 never reaches her.

The risk instead is staleness: a summary of a ticket that has moved on. So Tomás promises freshness:

• Agents never wait for a summary.
• It covers every message except at most the newest 4, plus any that arrived in the last 30 seconds.

The 30 seconds is the 23-second worst case plus headroom, and an assumption until measured.

Before launch, Tomás runs the summarizer over 200 old tickets, logging usage and the time of every call. That check costs about $1, and its numbers replace the assumptions.

Levers that come later

The estimate uses list prices. Later lessons lower them:

• Prompt caching: every refresh resends the same system prompt and the same older messages, and a cache read costs a tenth of the input price on Haiku 5.5 ^{1PricingAnthropic, Claude API documentation}. See Caching for LLM apps.
• Batch processing: work that can wait gets 50% off ^{1PricingAnthropic, Claude API documentation}. See Batch and background processing for LLM jobs.
• Model routing: a cheap model by default, a stronger one only for tickets that need it. See Model routing.
• Shorter output: asking the model to think less cuts both cost and the 8 seconds of thinking. See Reasoning effort.

Whether Haiku’s summaries are good enough is a question for Choosing a model.

Try it yourself

Time one small call and read its usage. It costs at most about $0.0005 on Claude Haiku 5.5. If you don’t have an API key, skip this.

curl -s -o reply.json -w "total: %{time_total} s\n" https://api.anthropic.com/v1/messages \
  -H "content-type: application/json" \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2026-10-21" \
  -d '{"model": "claude-haiku-5-5", "max_tokens": 1000, "system": "Summarize the support ticket in exactly three lines.", "messages": [{"role": "user", "content": "Ines: My order 7731 arrived without the charger.\nOmar (agent): Sorry. I will send one today.\nInes: It is now a week later and nothing has come. Can you check?"}]}'
python3 -c 'import json; print(json.load(open("reply.json"))["usage"])'

The output looks like this; the numbers here are illustrative and yours will differ:

total: 2.4 s
{'input_tokens': 85, 'output_tokens': 170, 'output_tokens_details': {'thinking_tokens': 110}, ...}

What to look for: output_tokens ÷ total time is a rough, low estimate of output speed, since the total includes time to first token. Run it five times; the spread between runs is why the budget has a worst case.

Where this shows up

• Design: a ticket summary feature turns this budget and promise into a design: when to refresh, where to store summaries, what to show when one is stale.
• Design: an AI customer support platform runs many calls per ticket, so cost per task decides which features are affordable.

The next lesson designs the feature that has to keep this budget and this promise.

Common mistakes

• Pricing the call, not the task. Count first summaries, refreshes and retries.
• Using one average ticket. Long tickets get more calls and bigger calls.
• Leaving thinking out. It is billed as output and is most of the wait.
• Promising speed for background work. Promise freshness, and measure it.

Check your understanding

Summary