Module 05 · Lesson 6 of 16Free

Hybrid search: combining keywords and vectors

Level

Intermediate

Topic

RAG: answering from your own data

Reading time

17 min

Order 48213 is in the index, and search misses it

Deskora is a help desk app. A shop’s support team uses it to answer customer emails from one shared inbox. Each customer problem becomes a ticket: one customer’s whole conversation with the support team.

Priya is a support agent at a kitchenware shop that uses Deskora. When she opens a ticket, a “Suggested answers” panel shows the three most similar resolved tickets from her shop’s 40,000.

A new ticket arrives from Ngozi: “Where is order 48213?”

The panel runs semantic search: it compares the meaning of Ngozi’s question with every resolved ticket. It returns three tickets about other customers’ late orders.

The ticket Priya needs is in the index. On Monday Ngozi wrote about the same order, and her colleague found the parcel held at the depot because of a wrong house number. That ticket contains “48213” twice. A keyword search for 48213 finds it first.

Tomás, the Deskora engineer who builds the AI features, can’t just switch to keyword search. Yesterday another customer wrote “My parcel never came”, and the right answer was the help-center article “Delivery not received”. They share no words. Keyword search missed it; semantic search put it first.

This lesson answers one question: how does one search find both the exact match and the match in meaning?

Every ticket, score and recall figure in this lesson is a made-up example.

Each search reads a different thing in the text

Semantic search turns the query into an embedding, a list of numbers that stands for its meaning. It then returns the stored tickets whose embeddings are closest, usually by cosine similarity. Embeddings shows how those numbers capture meaning, and Semantic search builds the panel on them. In this lesson, “vector search” means the same thing.

Keyword search matches the exact words. It keeps an inverted index: for each word, the list of tickets that contain it. It ranks matches with BM25, a formula that gives rare words more weight, gives less extra credit for each repeat, and adjusts for document length. BM25 explains the formula.

Rarity is why keyword search wins on 48213. That number appears in 2 of 40,000 tickets, so a match on it scores high. The word “order” appears in most tickets, so it adds little.

How the two methods do on four kinds of Deskora query:

The pattern has a cause on each side. An embedding model is trained to place texts with the same meaning close together. A code like “E-1042” is a few tokens with no meaning of their own, so in a 300-word ticket they barely move the vector. Anthropic’s retrieval guide gives the same warning: embeddings can miss an exact match on an error code ^{2Introducing Contextual RetrievalAnthropic, Anthropic, 2024}.

Keyword search has the opposite blind spot. It compares strings after light clean-up, such as lowercasing and cutting “parcels” to “parcel”. To it, “parcel” and “delivery” are unrelated words.

Customers often mix both in one sentence: “Order 48213 never came” needs the number and the meaning.

Run both searches, then merge two ranked lists

Hybrid search runs keyword search and vector search on the same question, then merges the two ranked lists into one.

Step by step, for Ngozi’s ticket:

1. The app server sends Ngozi’s question to the embedding model and gets its vector back.
2. It runs the keyword search and the vector search at the same time. Each returns its top 20 tickets.
3. Both searches filter to Priya’s shop. A shop must never see another shop’s tickets, and the merge keeps whatever either list returns, so one unfiltered side would leak them. Filtering vector search by metadata covers the vector side.
4. The fusion step merges the two lists, and the panel shows the top 3.

Why fetch 20 when the panel shows 3? A ticket ranked 8th in both lists may deserve a place above one ranked 1st in only one list, and the merge can only promote tickets it has seen. Twenty is this scenario’s choice, not a rule.

The merge is the hard part, because the two scores don’t share a scale. Ngozi’s Monday ticket has a BM25 score of 14.2. “My order is a week late” has a cosine similarity of 0.86. BM25 has no upper limit, and its range shifts with every query. Deskora’s cosine scores mostly sit between 0.7 and 0.9. Add the raw numbers and BM25 decides every result.

Reciprocal rank fusion merges by position

Reciprocal rank fusion (RRF) ignores the scores and uses only each ticket’s position in each list:

The paper that introduced the method used k = 60 ^{1Reciprocal Rank Fusion outperforms Condorcet and Individual Rank Learning MethodsGordon V. Cormack, Charles L. A. Clarke and Stefan Büttcher, Proceedings of the 32nd International ACM SIGIR Conference, 2009}. Here are the two lists for Ngozi’s question, cut to 5 each:

Each ticket collects 1 ÷ (60 + rank) from every list it appears in:

The panel shows T-901, T-512 and T-640. Ngozi’s Monday ticket is first, and the meaning matches still fill the rest.

Within one list, rank 1 earns 0.0164 and rank 5 earns 0.0154: position matters little. Appearing in both lists matters a lot. The four tickets both searches found score about twice as much as the two only one found.

The fusion step is a few lines of Python:

from collections import defaultdict


def reciprocal_rank_fusion(
    ranked_lists: list[list[str]], k: int = 60, top_n: int = 3
) -> list[tuple[str, float]]:
    """Merge ranked lists of ticket ids, best first. Ranks start at 1."""
    scores: dict[str, float] = defaultdict(float)
    for ranked in ranked_lists:
        for rank, ticket_id in enumerate(ranked, start=1):
            scores[ticket_id] += 1 / (k + rank)
    merged = sorted(scores.items(), key=lambda item: item[1], reverse=True)
    return merged[:top_n]


keyword_ids = ["T-901", "T-377", "T-640", "T-245", "T-512"]
vector_ids = ["T-512", "T-118", "T-640", "T-901", "T-377"]

for ticket_id, score in reciprocal_rank_fusion([keyword_ids, vector_ids]):
    print(f"{ticket_id}  {score:.5f}")
# T-901  0.03202
# T-512  0.03178
# T-640  0.03175

It never looks at a BM25 score or a cosine similarity. So it needs no tuning, and it keeps working when Tomás swaps the embedding model and every cosine score shifts.

Where RRF loses the exact match

The same property cuts the other way. Suppose Ngozi’s Monday ticket had not been in the vector list at all. Its score would be 1/61, about 0.0164, below T-512, T-377 and T-640. It would fall to fourth, out of the panel.

In general, a ticket found by both searches within their top 20 scores at least 1/80 + 1/80 = 0.025. So if three or more tickets appear in both lists, an exact match found only by keyword search drops out of the panel. RRF also throws away the size of T-901’s lead: 14.2 against 6.1 counts the same as 6.2 against 6.1.

For order numbers there is a cheaper fix than any merge. They follow a known pattern, so the app server can spot one with a regular expression, look it up directly and pin those tickets above the merged list.

Weighted blending keeps the score gaps

The alternative to RRF is to blend the scores themselves. First rescale each list to 0–1, for example (score − lowest) ÷ (highest − lowest) within that list. Then combine: final = w × keyword + (1 − w) × vector, with a missing ticket counting as 0. With w = 0.5, T-901 gets 0.5 × 1.0 + 0.5 × 0.2 = 0.60 and T-512 gets 0.5 × 0 + 0.5 × 1.0 = 0.50. T-901 now leads by 0.10 instead of 0.0002, so the size of a gap survives. The price: the last ticket in each list always gets 0, however good it is, and w must be retuned whenever either search changes.

Measure each search on labelled Deskora queries

Tomás builds a labelled set: 200 real questions from the shop’s past tickets. For each one, Priya and a colleague mark which resolved tickets answer it. Each mark is a relevance judgment, and each question has about 2 right tickets.

He scores each search with recall at k: of the right tickets for a question, the share that appear in the top k results. He uses k = 3, because the panel shows three tickets. For Ngozi’s question, with 2 right tickets, a top 3 holding one of them scores 0.5. Search quality metrics compares recall with precision and NDCG.

An average over all 200 hides the story, so he splits them by type:

Illustrative recall at 3, not measured. Each “All” figure is the weighted average of the two rows above it: for hybrid, (150 × 0.81 + 50 × 0.88) ÷ 200 = 0.83.

Three things to read from it:

• Vector search alone beats keyword search overall (0.70 against 0.59), but on identifier questions it finds fewer than half the right tickets.
• Hybrid is close to the better method on each type, so it wins overall.
• On identifier questions, hybrid is slightly below keyword alone (0.88 against 0.90). That gap is the lone-exact-match loss shown above; pinning order-number matches would close it.

Rerun the same set after every change: a new embedding model, a new analyzer, a different k. Evaluating RAG builds this into a full test of retrieval and answers.

Where the keyword search runs

If the vectors already live in PostgreSQL with pgvector, the keyword half can live in the same database. PostgreSQL’s full-text search stores each ticket as a tsvector, a list of cleaned-up words with their positions. A GIN index on that column is PostgreSQL’s inverted index. The pgvector documentation suggests exactly this pairing for hybrid search, merged with RRF ^{3pgvector: open-source vector similarity search for Postgrespgvector contributors, GitHub}.

-- Keyword half: a tsvector column on the tickets table
ALTER TABLE tickets ADD COLUMN search_text tsvector
  GENERATED ALWAYS AS (
    to_tsvector('english', coalesce(subject, '') || ' ' || coalesce(body, ''))
  ) STORED;
CREATE INDEX tickets_search_text_idx ON tickets USING gin (search_text);

-- Top 20 by keyword, Priya's shop only
SELECT id, ts_rank_cd(search_text, query) AS score
FROM tickets, websearch_to_tsquery('english', 'Where is order 48213?') AS query
WHERE shop_id = 204 AND search_text @@ query
ORDER BY score DESC
LIMIT 20;

-- Vector half: 20 nearest by cosine distance, same shop
SELECT ticket_id FROM ticket_embeddings
WHERE shop_id = 204
ORDER BY embedding <=> $1
LIMIT 20;

The app server runs both queries at once and passes the two id lists to reciprocal_rank_fusion. websearch_to_tsquery accepts raw customer text without raising syntax errors ^{4Controlling Text SearchPostgreSQL Global Development Group, PostgreSQL documentation}.

Two limits come with it. First, PostgreSQL’s built-in rankers, ts_rank and ts_rank_cd, are not BM25: they use no information from the rest of the table ^{4Controlling Text SearchPostgreSQL Global Development Group, PostgreSQL documentation}. So a rare word like 48213 gets no extra weight for being rare. Second, websearch_to_tsquery joins the words with AND, so a long question can match nothing. Extensions that add BM25 to PostgreSQL exist, and a dedicated search engine is the other route.

In production, give each search its own timeout, and if one side is slow, show the other side’s list alone. Contextual retrieval helps both halves: the context added to each chunk feeds the keyword index and the embeddings alike.

Where this shows up

• Design: a support assistant over company docs: hybrid retrieval picks the tickets and articles the drafted reply rests on.
• Design: an AI search engine with citations: searches must find both named things and described ideas.
• Design: an AI customer support platform: order numbers and error codes arrive in every other message.

Hybrid search gets the right ticket into the merged list. Putting the single best one first, by reading the question and each ticket together, is the next lesson’s job.

Common mistakes

Check your understanding

Summary