Module 11 · Lesson 2 of 9Free

Tracing LLM calls: seeing every step

Level

Intermediate

Topic

Observability and operations

Reading time

17 min

A strange draft, and one log line

Deskora is a help desk app. A shop’s support team uses it to answer customer emails from one shared inbox. Each customer problem becomes a ticket: one customer’s conversation with the support team. For each ticket, Deskora’s drafting assistant writes a reply from the shop’s help-center articles and similar past tickets, and a support agent edits and sends it.

Priya is a support agent at a kitchenware shop on Deskora. Yesterday afternoon a customer, Ines, wrote that her casserole dish arrived with a cracked lid. The draft said: “Thank you for the photos of the cracked lid. We’re sorry it arrived like this.” Then it stopped, with nothing about the shop’s rule: damaged items must be reported within 14 days, and then Priya decides what to offer.

Priya flags it. Tomás, the Deskora engineer who builds the AI features, opens the draft worker’s log and finds one line:

2026-10-28 15:42:09 INFO draft_job done ticket=tkt_58812 shop=204 duration=9.1s

He can see that the draft exists. He can’t see which articles were retrieved, which prompt version ran, or what the model wrote before a guardrail (a check on a model’s input or output) changed it. Nor why the job took 9 seconds, when a short ticket usually takes about 5; if Priya opens a ticket before its draft is ready, she waits that long.

This lesson answers one question: how can Tomás see every step of one draft, with its inputs, outputs, time and cost, without copying every customer’s email into another system?

The tickets, timings, token counts and sizes in this lesson are estimates for this scenario. Prices are for Claude Haiku 5.5 as of October 2026.

A trace is one tree of spans per task

A trace is the record of one task, here one draft job, made of spans. A span is one timed step with a name, start and end times, a status (OK or error) and attributes: key–value pairs such as deskora.ticket.id = tkt_58812. Every span but the first has a parent, so they form a tree sharing one trace id.

Each step records what it needs to explain itself later:

Inputs and outputs appear as ids and hashes, not text; why comes later.

Tomás could write these records himself. Here is why he uses a standard.

OpenTelemetry: one format every tool reads

OpenTelemetry is an open standard, with libraries for most languages, for producing traces, metrics and logs. In Python, code calls the API package, opentelemetry-api, to create spans, and the SDK package, opentelemetry-sdk, decides which to keep and exports them. Spans leave the process over OTLP, OpenTelemetry’s protocol, to a tracing backend that stores them and draws each trace as a timeline.

A hand-written record per draft, like the one Debugging RAG designed for retrieval, has no timeline and misses any step nobody remembered to log. A standard adds three things:

• Libraries already speak it. The anthropic SDK and instrumentation for web frameworks and databases create spans that join your tree.
• One trace crosses services. The trace context travels in a traceparent header or a queue message, so the app server that queues a job and the worker that runs it share a trace (Observability: logs, metrics and traces covers this).
• The backend can change without touching the code that creates spans.

The SDK records a span for every call

The anthropic Python SDK, version 2.3.0, depends on opentelemetry-api and creates spans itself ^{1anthropic-sdk-pythonAnthropic, GitHub}. Until the application registers a tracer provider, those spans record nothing. Once one is registered, every SDK call gets one span: for client.messages.create, a client span named anthropic.messages.create. Its attributes use OpenTelemetry’s generative AI names:

Three behaviours matter when you read one ^{1anthropic-sdk-pythonAnthropic, GitHub}:

• Retries stay inside one span. There is no span per attempt. The span covers every attempt and the waits between them, and http.request.resend_count says how many times the request was sent again.
• Errors mark the span with status error and an error.type, such as overloaded_error or client_timeout_error.
• A streamed reply’s span ends when your code finishes reading the stream, and records gen_ai.response.time_to_first_chunk.

The SDK’s tool runner adds its own spans too. Lab: adding tracing to an agent traces a hand-written agent loop instead, and adds a span for each tool call itself.

Metadata or content: one choice on the client

The client option open_telemetry sets what the spans hold. traces.content_mode is "metadata_only" by default: the attributes above, no text ^{1anthropic-sdk-pythonAnthropic, GitHub}. With "content", each span also gets the system prompt, every input message and the reply, thinking included, as gen_ai.system_instructions, gen_ai.input.messages and gen_ai.output.messages, up to 750,000 bytes per span by default ^{1anthropic-sdk-pythonAnthropic, GitHub}.

The variable ANTHROPIC_OPEN_TELEMETRY_TRACES_CONTENT_MODE=content turns content on only for a client created without the option ^{1anthropic-sdk-pythonAnthropic, GitHub}. So set the mode in code, and no environment variable can switch customer text on.

Deskora’s spans wrap the SDK’s

The SDK starts its span from whatever span is current. start_as_current_span opens a span and makes it current until its with block ends, so a model call inside write_draft gets a child span of write_draft.

# pip install anthropic==2.3.0 opentelemetry-sdk==1.45.1
import hashlib
import json
from typing import Any

import anthropic
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter

provider = TracerProvider(resource=Resource.create({"service.name": "draft-worker"}))
provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)  # from here on, every SDK call records a span
tracer = trace.get_tracer("deskora.drafting")
client = anthropic.Anthropic(open_telemetry={"traces": {"content_mode": "metadata_only"}})


def sha256(value: Any) -> str:
    return hashlib.sha256(json.dumps(value, sort_keys=True).encode()).hexdigest()


def draft_job(ticket_id: str, shop_id: str, thread: str) -> str:
    with tracer.start_as_current_span("draft_job") as job:
        job.set_attributes({"deskora.ticket.id": ticket_id, "deskora.shop.id": shop_id,
                            "deskora.prompt.version": "draft-v7"})
        with tracer.start_as_current_span("retrieve") as span:
            sources = [("damaged#1@v4", "Report damaged items within 14 days.")]  # stand-in
            span.set_attribute("deskora.source.ids", [source_id for source_id, _ in sources])
        context = "\n".join(text for _, text in sources)
        request: dict[str, Any] = {"model": "claude-haiku-5-5", "max_tokens": 1024, "messages": [
            {"role": "user", "content": f"Help center:\n{context}\n\nTicket:\n{thread}"}]}
        with tracer.start_as_current_span("write_draft") as span:
            reply = client.messages.create(**request)  # the SDK's span opens inside this one
            draft = "".join(b.text for b in reply.content if b.type == "text")
            span.set_attributes({"deskora.request.sha256": sha256(request),
                                 "deskora.draft.sha256": sha256(draft)})
        with tracer.start_as_current_span("output_check") as span:
            promise = any(word in draft.lower() for word in ("refund", "replacement"))  # stand-in
            span.set_attribute("deskora.check.promise", promise)
        cost = (reply.usage.input_tokens * 0.10 + reply.usage.output_tokens * 0.50) / 1_000_000
        job.set_attribute("deskora.cost_usd", cost)  # Claude Haiku 5.5 prices, October 2026
        return draft


print(draft_job("tkt_58812", "204", "Customer: The lid of my casserole dish arrived cracked."))
provider.shutdown()  # sends the last spans before the program exits

The search and the check are stand-ins. The cost uses Claude Haiku 5.5’s prices as of October 2026: $0.10 per million input tokens and $0.50 per million output tokens, for prompts up to 100,000 tokens ^{3PricingAnthropic, Claude API documentation}. One run is about 60 input tokens and a few hundred output tokens: well under $0.001.

The exporter prints each span as JSON. Trimmed, the SDK’s span from one run (against a test stand-in, so the numbers are examples):

{
  "name": "anthropic.messages.create",
  "kind": "SpanKind.CLIENT",
  "parent_id": "0xda2e15e6318b08a2",
  "status": {"status_code": "OK"},
  "attributes": {
    "gen_ai.provider.name": "anthropic",
    "gen_ai.request.model": "claude-haiku-5-5",
    "anthropic.request.id": "req_011CExample",
    "anthropic.message.stop_reason": "end_turn",
    "gen_ai.usage.input_tokens": 52,
    "gen_ai.usage.output_tokens": 210,
    "gen_ai.usage.reasoning.output_tokens": 160
  }
}

parent_id is write_draft’s span id. In production, Deskora swaps ConsoleSpanExporter for OTLPSpanExporter from opentelemetry-exporter-otlp-proto-http, pointed at its backend with OTEL_EXPORTER_OTLP_TRACES_ENDPOINT. BatchSpanProcessor exports from a background thread, so a draft never waits for the backend.

The code records hashes where it could have recorded text. That is the first real decision.

Ids and hashes, not customer text

A span that holds the prompt holds the customer’s email. At Deskora’s 75,000 drafts made a day:

Size is the smaller problem: every one of those records would hold a customer’s name, email and address.

A hash of a short, guessable value, such as an email address, can be reversed by hashing guesses, so hash whole requests and outputs only.

Each copy also needs a data retention rule. Deskora keeps traces 30 days, long enough for most flags to arrive (assumed), and the debug store 7.

Keep every draft’s trace

Sampling cuts storage by keeping some traces and dropping the rest. The OpenTelemetry Python SDK’s default sampler, parentbased_always_on, keeps everything ^{2opentelemetry-pythonOpenTelemetry, GitHub}. The System Design observability lesson compares deciding at the start of a request with deciding after it ends; an LLM feature changes that comparison.

Reading Priya’s trace, step by step

Tomás ships tracing that evening and switches on masked capture for Priya’s shop. The next afternoon Priya flags a second draft like Ines’s: Bruno’s kettle arrived dented, and the draft thanks him for the photos and stops, after 9.1 seconds. Ines’s draft very likely failed the same way, but nothing recorded can prove it: evidence must exist before the incident. Bruno’s draft has a trace.

The fix is in retrieval, not the prompt: what happens when the reranker is slow, which Design: a support assistant over company docs set as a 250 ms timeout and a fallback to hybrid order.

The trace also gives the cost of the whole task, the unit that matters. Every call is Claude Haiku 5.5: input tokens × $0.10 per million + output tokens × $0.50 per million, thinking included ^{3PricingAnthropic, Claude API documentation}:

Without the regenerate it costs about $0.0010, so this failure made it about 70% dearer. Rerank and embedding calls aren’t counted; gen_ai.usage.reasoning.output_tokens gives the real thinking count.

Traces join the logs and the eval set

Log lines hold the detail each step wrote. A trace id on every line joins them to the trace. A logging filter reads it from the current span:

import logging

from opentelemetry import trace


class TraceIdFilter(logging.Filter):
    def filter(self, record: logging.LogRecord) -> bool:
        context = trace.get_current_span().get_span_context()
        record.trace_id = format(context.trace_id, "032x") if context.is_valid else "-"
        return True


handler = logging.StreamHandler()
handler.addFilter(TraceIdFilter())  # on the handler, so the SDK's own log lines get it too
handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s trace_id=%(trace_id)s %(message)s"))
logging.basicConfig(level=logging.INFO, handlers=[handler])

The SDK logs each retry, such as Retrying request to /v1/messages in 0.43 seconds, while its own span is current, so that line carries the job’s trace id too.

The eval set joins the same way. Debugging RAG turned each debugged flag into a labelled test case. Bruno’s ticket becomes one, scrubbed of personal data, tagged “rerank fallback, policy article missing”, with the trace id it came from. Eval runs call the same draft_job code with deskora.run = "eval" and the case id, so a failing case opens to a trace like a production one. Notes in Error analysis can cite trace ids.

Where the spans go: backends

Any backend that accepts OTLP can store these spans. Two examples, from their own documentation:

• Langfuse receives traces on its /api/public/otel endpoint, with Basic auth from a public and a secret key, over HTTP (JSON or protobuf; not gRPC yet). It aims to follow OpenTelemetry’s generative AI conventions, and its docs cover self-hosting ^{4OpenTelemetry (OTEL) for LLM ObservabilityLangfuse, Langfuse documentation}.
• Arize Phoenix accepts traces over OTLP and isn’t tied to one model vendor or framework ^{5Overview: TracingArize AI, Phoenix documentation}.

Pick by where the data lives, whether it reads the gen_ai attributes, and what your volume costs there.

Where this shows up

• Design: an AI customer support platform: one trace per ticket across drafting, the resolution agent and chat.
• Design: a voice ordering assistant: spans show where a spoken reply’s time goes.
• Design: scaling a chat assistant to a million users: what to keep per trace at that volume.

A trace explains one draft someone flagged. It can’t say how many went wrong that nobody flagged; Monitoring quality in production takes that on.

Common mistakes

Going deeper

Check your understanding

Summary