Module 12 · Lesson 1 of 9Free

Prompting, RAG or fine-tuning: choosing the right fix

Level

Intermediate

Topic

Fine-tuning and model customization

Reading time

16 min

Three problems land in one week

Deskora is a help desk app. A shop’s support team uses it to answer customer emails from one shared inbox. Each customer problem becomes a ticket: one customer’s whole conversation with the support team. Deskora’s AI features draft replies, summarise long tickets and route each new ticket to the right queue.

Tomás, the Deskora engineer who builds those features, gets three reports in the same week:

1. A stale rule. Priya, a support agent at a kitchenware shop, sees a draft that says free delivery starts at $60. Her shop lowered it to $40 last Monday.
2. The wrong voice. Ines runs a small tea shop that writes to customers like friends. Its drafts read like a bank letter: “Dear valued customer, we sincerely apologise for any inconvenience.”
3. A server rule. A pharmacy chain wants to move its support onto Deskora. Its customers mention medicines, so its contract says ticket text must never leave its own servers. That includes the router, the model call that puts each new ticket in a queue.

Kwame, the Deskora engineer on call, has one answer for all three: “Fine-tune it.”

This lesson answers one question: which fix fits each problem, and what does each one really cost?

The $40 threshold, the tea shop and the pharmacy are this lesson’s scenario. The pharmacy’s 9,000 tickets a day and every time estimate below are assumptions. Deskora’s 120,000 tickets a day and 60,000 drafts a day are facts used across this course.

Each fix changes a different part of the call

Every one of Deskora’s AI features is a model call: text goes in, the model writes text back. There are three places Tomás can change what comes out.

• The prompt holds instructions and examples. You change it by editing text, and the next call uses it. Few-shot prompting showed how a few examples carry a style that’s hard to put in words.
• Retrieval finds passages, such as a help-center article, and puts them in the prompt at the moment of the call. This is RAG, short for retrieval-augmented generation. Change the source and the next search finds the new text.
• Fine-tuning trains an existing model further on your own examples. It changes the model’s weights: the billions of numbers that decide what it writes. What it learns is a habit that applies to every call, whatever the prompt says.

Training teaches habits, not reliably facts

Many engineers believe fine-tuning is how you teach a model your company’s knowledge. It sounds right: training is how the model learned everything else.

Research says otherwise. One study compared retrieval with fine-tuning on raw text about a topic. Retrieval consistently did better, both for facts the model had seen before and for new ones ^{1Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMsOded Ovadia, Menachem Brief, Moshik Mishaeli, Oren Elisha, arXiv, 2023}. A second study found that models learn examples with new facts much more slowly than other examples. As those facts are learned, the model makes things up more often ^{2Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, Jonathan Herzig, EMNLP 2024, 2024}. Its authors conclude that models gain facts mainly in pretraining, and that fine-tuning mostly teaches them to use what they know ^{2Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, Jonathan Herzig, EMNLP 2024, 2024}.

So fine-tuning is good at habits: a format, a tone, one narrow skill done the same way every time. Facts that belong to one customer, or that change, belong in the prompt.

No Claude model can be fine-tuned today

As of October 2026, Anthropic offers no customer fine-tuning for current Claude models. The one route was Claude 3 Haiku on Amazon Bedrock. That model reached end of life in most Bedrock regions, the US ones among them, on 10 September 2026. In its last regions it is a legacy model, and Bedrock allows no new fine-tuning jobs on legacy models ^{3Model lifecycle (Legacy)Amazon Web Services, Amazon Bedrock User Guide}.

So for Tomás, fine-tuning means an open-weight model: one whose weights you download, train and then serve yourself. That is the trade Hosted and open-weight models described, plus a training step.

Three questions pick the fix

Tomás uses the same three questions for every problem, in this order.

1. Is a fact missing, private or out of date? Use retrieval, or put a short fact in the prompt. Training won’t hold it reliably.
2. Can you describe the behaviour, or show it in a few examples? Use the prompt. It costs input tokens and an afternoon.
3. Is it one narrow task with a stable format, which a smaller model you run must do? Then fine-tuning fits. The usual reasons are volume past what the API costs, or a rule that keeps text away from any provider. Often the training labels come from a large model, which is Distillation.

The fixes combine. A fine-tuned model still reads a prompt, and a prompted model still gets retrieved passages. The questions only decide where each part of the problem goes.

The $40 rule is a stale source

Tomás starts with question 1. Tunde, a customer, asked whether his $45 order ships free. The draft said no, free delivery starts at $60.

Every draft cites its sources, so Tomás opens the citation. It points to the shop’s “Delivery times and costs” article, last edited in March. Hana, the shop’s manager, announced the $40 threshold on the shop’s home page and in her team chat. She never edited the help-center article.

Retrieval worked. It found the right article, and the article was wrong. Hana edits it, and Deskora’s update job re-indexes it within minutes, as built in Keeping a RAG index fresh.

No model changed. The fix was the source.

The tea shop’s voice fits in a prompt

The tea shop fails question 1: no fact is wrong. It passes question 2. Ines can describe her voice, and her past replies show it.

Tomás adds a style note to the tea shop’s settings, plus three of its own sent replies as examples. Deskora puts both in the drafting prompt, before the ticket. He picks examples that make no promises. The model copies everything examples share, and drafts must never promise a refund or replacement; the support agent decides.

Before release, the change runs Deskora’s drafting eval set: about 300 labelled tickets that every prompt change is tested on. That way a playful tone doesn’t cost accuracy elsewhere. The whole fix is an afternoon of work.

Fine-tuning for tone would mean moving drafting off Claude Haiku 5.5 onto an open model Deskora serves. It would also need hundreds of replies per shop, and a new training run whenever a shop changes its voice. That trade can make sense for one fixed house style at very large volume. It doesn’t for 1,500 shops with 1,500 voices.

Try it yourself

Send one customer email to Claude Haiku 5.5 twice: once with the tea shop’s style in the system prompt, once without. One call costs under $0.00025: about 300 input tokens × $0.10 per million, and at most 400 output tokens, thinking included, × $0.50 per million. If you don’t have an API key, skip this.

curl https://api.anthropic.com/v1/messages \
  -H "content-type: application/json" \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2026-10-21" \
  -d '{"model": "claude-haiku-5-5", "max_tokens": 400, "system": "You draft replies for support agents at Kettlewick Tea, a small tea shop. Never promise a refund, replacement or discount; the agent decides that. Style: warm and playful, first names, short sentences, at most one light joke. Sign off: Cheers, the Kettlewick team. Example reply: Hi Ama! Oh no, a dented tin is a sad start to tea time. Could you send us a photo of the dent? We will look at it and get back to you today. Cheers, the Kettlewick team", "messages": [{"role": "user", "content": "Draft a reply to this email. Hello, my order 52817 arrived and the jasmine tea tin is crushed. The tea spilled everywhere. What can you do? Ravi"}]}'

An example reply text from one run, shortened; yours will differ:

{"type": "text", "text": "Hi Ravi! A crushed tin and jasmine everywhere is not the calm cup we had in mind. Could you send us a photo of the tin and the box? ... Cheers, the Kettlewick team"}

What to look for:

• The reply follows the voice of one example and the style note. Nothing about the model changed.
• Delete the "system" field and run it again. The reply usually turns formal and generic.
• Check that neither version promises a new tin. If one does, the prompt needs that rule stated more plainly.

The pharmacy’s router is a job for fine-tuning

The pharmacy passes questions 1 and 2. Deskora’s router prompt already describes the six categories well, and Claude Haiku 5.5 follows it. But the call can’t leave the pharmacy’s servers, and Claude’s weights can’t be downloaded. So the router must be a model that runs in the pharmacy’s own data centre.

Tomás’s candidate is Qwen3-1.7B, a small open-weight model with 1.7 billion parameters, under the Apache 2.0 licence ^{4Qwen3-1.7BQwen team, Hugging Face model card}. It fits one modest GPU. Routing is one narrow task with a fixed answer format: a category, an urgency and an order number. That is question 3.

This doesn’t save Deskora money

Kwame takes it one step further: “If the small model routes as well as Haiku, move all 120,000 daily tickets onto it.”

Self-hosting open models already did that sum. Routing on Claude Haiku 5.5 costs about $25 a day. Two GPUs plus staff time cost about $62 a day, and self-hosting breaks even near 300,000 tickets a day. So Deskora’s own router stays on Haiku. The pharmacy pays for its rule, not for savings: its 9,000 tickets a day would cost about $1.86 a day on Haiku.

The training run is the smallest cost

The training run is the line people budget for. For a model this small it is a few GPU-hours, an assumption for this lesson. The costs that come after it are larger.

Preparing data for fine-tuning and Evaluating a fine-tuned model cover the two largest rows.

Drift is the cost that surprises people most. A prompt change takes hours. A fine-tuned model turns each change into a small project.

Where this shows up

• Design: an AI customer support platform: the same choice between prompt, retrieval and training, made for each feature.

Tomás has chosen to fine-tune. The next lesson looks at what a training run actually changes inside the model.

Common mistakes

• Fine-tuning to add facts. Facts learned in training are unreliable and can raise the rate of made-up answers. Put them in a source the model retrieves.
• Skipping the prompted baseline. Without a score for the prompted model, you can’t tell whether training helped.
• Budgeting only the training run. Data, evals, retraining and serving cost more than the GPU-hours.
• One shared model for one customer’s data. Weights trained on one shop’s replies affect every shop that uses them, and can repeat parts of that text.

Check your understanding

Summary