Module 15 · Lesson 3 of 9Free

AI system design interviews: a framework

Level

Intermediate

Topic

AI engineering interviews

Reading time

18 min

Minute 24 of 45. The prompt was: “Design an assistant that answers customer questions from a company’s help center.” You have drawn a load balancer, three app servers, PostgreSQL and a Redis cache.

The interviewer asks: “A customer asks if they can return opened headphones. How do you know the answer is right, and what did that answer cost?” You have no number for either.

The boxes were the design of any web app. This lesson answers one question: what does an AI system design interview test that a classic one doesn’t, and how do you spend 45 minutes showing it?

What the interviewer is trying to learn

Companies describe design rounds in general terms. Atlassian’s handbook, for example, says its 60-minute session wants to see how you explore a problem and which questions you ask.^{1Atlassian engineering interview handbookAtlassian, Atlassian Careers} It also says interviewers may add new constraints, such as reliability or cost, and watch how you adjust.^{1Atlassian engineering interview handbookAtlassian, Atlassian Careers}

An AI design round keeps all of that. It adds four things a classic round rarely reaches. This list is this lesson’s view of the gap, not any company’s scoring sheet.

A token is the unit a model reads and bills, roughly three quarters of an English word. Because output is non-deterministic, “it worked when I tried it” proves little. So the interviewer listens for three things: a price per task, a way to measure quality, and a list of what the system must never do.

The four differences tell you what to cover. The next question is in what order, and for how long.

Eight steps on one clock

Use the same skeleton for every AI design prompt. The timings are this lesson’s practice default for a 45-minute round, not any employer’s rule. The System Design course’s 45-minute interview framework explains how to work with checkpoints instead of a rigid timer; this lesson adds the AI steps.

Evals and safety get 13 minutes, because a classic answer leaves them out. Now walk the opening prompt through the steps.

Clarify: five questions that change the design

Ask only questions whose answers change a box or a number. If the interviewer leaves one to you, state your assumption and write it down.

The last row is a non-functional requirement, stated as a latency percentile. These are the interviewer’s numbers for this example, not facts about any real company.

Then cut scope aloud: “Out of scope for now: order lookups, account data and other languages.” That buys time for evals later.

Estimate per task, not per server

The unit of cost is one completed conversation, not one API call. This estimate counts the answer calls only. It leaves out the query embedding and any safety classifier, which are small; say so on the board. The formula and its habits come from estimating the cost and latency of a feature.

Prices are for Claude Haiku 5.5 as of October 2026: $0.10 per million input tokens and $0.50 per million output tokens, for prompts up to 100,000 tokens.^{2PricingAnthropic, Claude Platform documentation, 2026} Haiku 5.5 thinks before it answers. Thinking tokens are billed as output, so each case assumes a number of them.

That is a number that changed a plan: the account tier, not the server count. The latency target gets one more line: retrieval should take about 200 ms (an estimate), which leaves the rest of the 2 seconds for the model’s first words, so stream the reply.

Draw one picture and trace one question

Draw the request path on the right, the ingestion path on the left, and keep the numbers box in a corner. Skip load balancers and replicas unless asked; at 2 calls a second they decide nothing.

The traced question: the customer asks about opened headphones (1). The chat API sends the question to the retriever (2). It searches with hybrid search, keywords plus embeddings, and returns 5 chunks from the returns article (3). The model gets the rules, the chunks and the history, and answers from them (4). The output check confirms the answer makes no promise of a refund (5). The reply streams back (6).

This design is retrieval-augmented generation (RAG). The next step is to defend that choice with numbers.

Decision: where the articles enter the prompt

The model choice gets one sentence: “Claude Haiku 5.5, because the task is short answers from given text and the latency target is tight; I’d test a larger model on the eval set before ruling it out.” That sentence leads straight to evals.

Evals: say how you would know it works

An eval is a repeatable test of output quality on fixed examples. Put five lines on the board:

1. Data: 300 real past customer questions, each labelled with the article that answers it and whether the right behaviour is to answer or hand off. Add attacks and refund demands.
2. Retrieval, in code: is the right article among the 5 chunks? Score it separately, so a bad answer can be traced to search or to the model.
3. Answers: is the answer a grounded answer, every claim supported by a chunk? An LLM judge grades this, checked against people’s grades on a sample.
4. Repeats: run each case 3–5 times and report rates, because one run proves little. Example gate: 95% grounded and zero refund promises.
5. Online: hand-off rate, thumbs down, and customers who contact support about the same issue within a day.

Evaluating RAG teaches these metrics. For the follow-up questions about them, see discussing evals and trade-offs under pressure.

Safety: what the assistant must never do

Start with a list, then give each item a mechanism. A rule that lives only in the prompt is a hope, not a control.

The first row is prompt injection: input text that tries to act as instructions. The fallback for everything is a hand-off: the conversation becomes a ticket for a person. Design: a safe customer-facing assistant builds a full version of this table. It also weighs the cost of a check against the 2-second target: streaming shows words sooner, but a reply checked only after it is shown can’t be taken back.

Cost and scaling: which limit breaks first

Multiply the numbers box by 10×. At 200,000 conversations a day, the busiest minute has about 1,250 calls and 5.2 million input tokens. That passes the Start tier’s 1,000 requests and the Build tier’s 5 million input tokens a minute; the Scale tier allows 10,000 and 10 million.^{3Rate limitsAnthropic, Claude Platform documentation, 2026} The database is still not the problem.

Name each lever in one clause. Prompt caching on the system prompt and repeated context helps twice, because cache reads don’t count toward the input-token limit on most current models.^{3Rate limitsAnthropic, Claude Platform documentation, 2026} Caching answers to common questions is the other lever, covered in caching for LLM apps.

Close on trade-offs: RAG over long context for the 30× cost gap, Haiku 5.5 until the eval set says otherwise, hand-off over guessing. Then say what you would build next.

What strong answers cover

Score your practice answers with this list.

If you have built something similar, describe what you measured, not only what you built. This course builds one for Deskora, a made-up help desk app: Design: a support assistant over company docs.

Common mistakes

• Quoting prices and limits as permanent. Say “as of my last check, about $0.10 per million input tokens; I’d confirm it.” The method matters more than the digit.
• Promising repeatable output with sampling settings. On Claude API version 2026-10-21, temperature returns a 400 error. Consistency comes from clear prompts, structured output and evals over repeated runs.
• Designing an agent when a fixed path works. Retrieve, answer, check is a workflow. Add tools that act only when a requirement needs them.

A full mock interview on a similar prompt, an HR policy assistant, with dialogue and deep dives, is the mock interview for a RAG assistant. The next lesson moves from the whiteboard to the keyboard: live coding against a model API.

Check your understanding

Summary