We are currently implementing semantic search for the 18.0.0 release of OpenProject, which will go live in December. The incredibly skilled AI team at OpenProject, which we collaborate with, has thought about almost every scenario that could arise when an existing product starts to use semantic search.
But this is not the norm. Many clients we have worked with have failed with their first, seemingly simple, transition to semantic / combined search.
So here it is: The "best of" of mistakes you can make when setting up semantic search!
No. 1. Assume your current provider and model will always be around
Say you experimented with embedding models, and they add great value to your search. You used OpenAI's text-search-davinci-doc-001. Everything's fine, but suddenly OpenAI announces that they are deprecating text-search-davinci-doc-001.
Now you have two choices: Do a hard switch and hope for the best, or do a soft migration.
The risk of a hard switch
We are used to assuming that a newer model is always better than an older one. This does not have to be true. If you are working with semantic search on code (like we do at bitcrowd), you might realise that the performance of some old model is unrivalled.
Changes in context length might also mean that your content is not retrieved as effectively as before.
The benefits of a transition
If you want to avoid those side effects, it's a good idea to link your embeddings to the environment they are created in: If you only store the embedding vectors, you lose the information on how they were created.
Recording the information about the inference environment gives you two things:
- Reproducibility
- The ability to roll back once things went wrong with a new provider.
| document_id | vectors | embedding_environment_id |
|---|---|---|
| 13 | ... | 23 |
| 13 | ... | 24 |
| 14 | ... | 23 |
| 14 | ... | 24 |
with embedding_environment like this:
| id | embedding_model_id | provider_id | .... |
|---|---|---|---|
| 13 | 378 | 1 | ... |
| 14 | 379 | 1 | ... |
Now you can have two (or more) semantic indices. So you can build and test the new semantic index with the new model / new provider and switch once you are confident everything works.
And if your favourite model ceases to exist, you can benchmark several others against each other.
No. 2. Measure Performance manually
If you don't have a test suite to benchmark your embedding model / provider setup, you need to test manually. And when you test manually, you might not test thoroughly once a model change does not fit your personal schedule. You only test one, two, three of your favourite search queries and it becomes very easy to lose track: Is that one record you expect to get as a result not there any more because it changed, or because your search is now broken?
You lose time in investigating, and you test less next time. But benchmarking should be fun!
So much fun that you love to do it, even if you do not need to change anything or have little time.
MTEB
One of the most impressive frameworks for search performance is the Massive Text Embedding Benchmark (MTEB), which is famous for the Hugging Face model leaderboard. MTEB scores embedding models across task types such as retrieval, classification and clustering, in many languages.
Some additional reading you might find useful:
- Evaluation Metrics for Search and Recommendation Systems – precision, recall, MRR, MAP and nDCG explained with small examples
- Evaluation in information retrieval – the chapter from Manning, Raghavan and Schütze's Introduction to Information Retrieval, free online
- Creating judgement lists with Quepid – a hands-on guide to building a judgement list with human raters
The core of performance evaluation:
- Precision: of the results you show, how many are relevant? Ten results, six of them useful: precision 0.6.
- Recall: of all the relevant items that exist, how many did you find? Eight relevant work packages, six of them in your results: recall 0.75.
The two pull against each other: show more results and recall goes up, while precision usually goes down. For a RAG system, you would go for more precision. Here, where we actually serve a human, we can optimise for recall.
The catch
Precision and recall work well if you know all "good" and "irrelevant" results, or, better still, have a score for each hit. Synthetic benchmarks therefore try to control one or the other:
- They bury a piece of information that is not present anywhere else in the document collection and ask for it, so, again, the result set is known. This "needle in the haystack" procedure has the benefit that the number of synthetic results is known.
- They generate specific search terms for a page so it must appear in the result set if the search is any good. Synthetic questions work without changing the document collection, which is great in production environments, but they might yield more (or better) results than only the page they were written for.
So how to evaluate the result set of a synthetic question?
- You can, of course, skim through the results yourself and score them
- You can use a separate LLM to do that operation for you (LLM as a judge)
- You can use a reranker and see if your target results rank significantly high.
No. 3. Use A Reranker
Ok, now you have two streams of search results. The conventional search your users have learned to cope with, and your shiny new semantic search. But it soon turns out that both have their strengths and weaknesses.
How do we achieve the best results? Use a reranker!
Wot Reranker is?
A reranker takes the results your search has already found and sorts them again, this time looking at each result together with the query. The first search has to be fast across the whole collection, so it only compares precomputed vectors or keywords. The reranker sees just a short list of candidates, so it can afford a much closer look.
There are two main types:
- Cross-encoders: a small model reads the query and one result at the same time and returns a relevance score. That's far more precise than comparing two embeddings computed separately, but it costs one model call per result.
- LLM rerankers: you hand the query and the candidates to a large language model and ask it to judge or order them. They're the most flexible, since you can describe in plain words what "relevant" means for you, and also the slowest and most expensive.
Not to be confused with rank fusion, such as Reciprocal Rank Fusion (RRF). It merges your keyword and semantic result lists purely by position, with no model involved. That's cheap and often good enough, but a result that both lists ranked low stays low.
In No. 2 we used a reranker to evaluate the quality of our synthetic questions and it worked well. So why not make it a habit and use a reranker in our search system itself?
The Reranker
Using a reranker has two limitations:
- It can only happen after all results have arrived. So it adds a lot more cycles for spin up, connection, result transfer ... This can lead to a significant lag that the user feels.
- It needs to operate on a limited set of results (that's the top N in the diagram above). Often, these systems use a "top k" filter.
What's Top-k and Top-p?
Top-k keeps a fixed number of results: the k with the highest scores, say the best 50 from each search. It's simple and predictable, but it doesn't care how good those results are. A query with three relevant work packages still gets padded to 50, and one with 200 gets cut off.
Top-p comes from how LLMs pick their next word (nucleus sampling). Instead of a fixed count, you turn the scores into shares that add up to 100%, then keep the best results until together they cover p, say 90%. When one result clearly stands out, the list stays short; when many look equally good, it grows. The catch: the scores have to mean something across results, so a reranker's scores tend to be a better fit than raw keyword or vector scores.
In practice, top-k suits what goes into the reranker, because it caps the cost. Top-p suits deciding how much of the reranked list to show.
While our classic search will not return any results if its search term does not match, top-k always returns a result, however ridiculous its matches might be.
RRF, the charming alternative
Reciprocal Rank Fusion (RRF) merges result lists using nothing but each item's position. Every list awards an item 1 / (k + rank) points, with k usually set to 60, and the points add up. A work package that ranks well in both keyword and semantic search climbs to the top. Since RRF never looks at the raw scores, you don't have to make keyword and vector scores comparable first. And there's no extra model call: it's plain arithmetic on two lists you already have.
A reranker is no mistake
But bringing one in before you checked the other alternatives might be. It is a more costly operation than calling an embedding model, and it has some latency costs. So you might want to be sure you need one. So try things like RRF first, check them against your benchmark from No. 2, and add a reranker only if the numbers say you need one.
No. 4. Treat Semantic Search as a filter
The subtle difference between search, filter and ordering
Search decides what counts as a match for the query. Filtering applies hard yes/no rules on fields such as project, status or permissions: a work package passes or it doesn't, whatever the query says. Ordering only sorts what's left, by relevance or by a field like date; it never adds or removes anything.
Keyword search really matches: no shared terms, no result. Semantic search doesn't: every item has some similarity to every query, so on its own it can only order. A similarity cutoff looks like a fix, but the right value shifts with every model and query.
So use semantic search to sort, and keep the yes/no decisions in filters.
The subtle catch
This is especially relevant if you offer the download of search results: Your users might expect more results than the 327 deduplicated results left over from the combined 600 or so search results.
No. 5. Don't Monitor your embedding endpoints
If your embedding model provider is down, your search cannot compute the embedding vectors for your semantic search. That might not sound too scary, but it might confuse RRF so that even the classic search might perform less well.
What is worse: If your calls to update existing embeddings fail, your semantic index will either become outdated or erode.
The remedy
The best probe is to listen to errors from the embedding calls. The good thing: Forwarding the error messages to a decent escalation mechanism comes for free as you need the embedding anyway. But those only fire when the system is used. Failed embedding column updates should retry / be retry-able once the problem is fixed.
Therefore, you should monitor the availability of your provider in general and your chosen model in particular with probes that run every few hours. This might give you some advance warning if your user base has a very good sleep and the model provider fails at night.
You should also subscribe to the newsletter / feed from your provider and scan especially for the names of the models you use. That way, you catch that end-of-service announcement way ahead of time. We did not do that for our RAG project Exmeralda and were therefore surprised twice :)