The search measurement ladder: nine rungs from 'search is on' to 'we know it works'
A nine-step way to place a search or RAG system by what it can actually decide: from nothing measured, through a search log and labels, to statistics and live tests. TensorOpt's framing, not an industry standard.
Anyone who runs search can say whether it is switched on. Saying whether it works takes more. Between those two statements there is a sequence of steps, and each step lets you make one kind of decision and not the next. This post lays the steps out as a ladder of nine rungs, so you can see which rung your system stands on and what climbing one more would give you.
One caveat before the rungs. The ladder is TensorOpt's framing, not an industry standard. It organises practices that are well established individually, such as search logs, relevance labels, agreement statistics and significance tests, into one order. This post makes no claim about where most systems sit on it.
Why misses are invisible
People judge search by what the screen shows them. A result that is on the page can be judged good or bad. A result that should have been on the page and is not leaves nothing to judge. That asymmetry is the reason search quality is so easy to overestimate.
The classic demonstration is Blair and Maron's 1985 study of a legal-discovery system (Blair and Maron, 1985). The system searched a large body of case documents with Boolean queries, and the lawyers using it believed they were finding most of the relevant material. When the researchers checked against documents found by other means, the lawyers had found far fewer than they thought.
Take the caveat seriously: that was a Boolean full-text system used for litigation, not a modern ranked or semantic search engine, and it says nothing about how well any particular system works today. The lesson that carries over is narrower. People cannot see what search failed to return, so confidence without measurement is not evidence.
The nine rungs
The rungs fall into four groups: Observe (1 to 2), Label (3 to 5), Decide (6 to 7) and Connect (8 to 9). For each rung, this section says what exists there, what you can decide, and the trap that catches teams standing on it.
Observe: rungs 1 and 2
Rung 1: search is switched on. Search runs and users can type into it. Nothing is measured. The only decision available is whether to start measuring. The trap is reading silence as success: no complaints reach the team, so search is assumed to be fine, when the people it fails may simply leave without saying anything.
Rung 2: a search log. There is a record of queries and the results that were shown for them. Top queries and zero-result queries become visible for the first time. At this rung you can decide which queries matter most and which ones return nothing, and you can pick the first things to look at. The trap is treating the log as a verdict. A log shows what users typed and what they were shown. It does not say whether what they were shown was any good, and the most frequent queries are not always the ones failing.
Label: rungs 3 to 5
Rung 3: labelled queries. A sample of logged queries has relevance labels, meaning someone has judged how well each result answers the query. Now precision, recall and nDCG can be computed. You can decide whether the sampled queries are served well and where the weakest ones are. The trap is a number that looks exact and is not. A single person's labels, or a sample made up only of popular queries, produces a figure with decimals that says little about search as a whole.
Rung 4: labeller agreement. Agreement between labellers is measured, so you know how reliable the labels are. At this rung you can decide how far to trust a score, and whether the labelling guidelines need rewriting. The trap runs in both directions. Low agreement can mean the guidelines are unclear, or that the task is ambiguous. High agreement does not prove the labels are right, because two labellers can share the same misunderstanding of what the user wanted.
Rung 5: an LLM judge, checked against human labels. A language model scores results at a scale people cannot match, and its scores have been compared with human labels so you know how far to rely on them. This rung lets you decide about many more queries than hand labelling allows. The trap is an unchecked judge. A judge that was validated once, on easy cases, can drift or carry its own bias, and a score from an unchecked judge is an opinion at scale.
Decide: rungs 6 and 7
Rung 6: configurations compared by average scores. Two or more configurations are scored on the same queries and compared by their averages. You can decide which configuration looks better. The trap that defines this rung: two averages can differ by chance. A different sample of queries could reverse the order, and a bare average gives no way to tell.
Rung 7: confidence intervals, paired tests and multiple-comparison control. Configurations are compared with confidence intervals around each score, tests that pair the two systems on the same queries, and a correction when many comparisons are made at once. At this rung you can decide whether a difference is likely to be real. The trap is subtler. Compare enough configurations and one will look significant by luck, which is what the multiple-comparison control is for. And a difference can be real and still too small to matter, so significance answers one question, not all of them.
Connect: rungs 8 and 9
Rung 8: search metrics tied to outcomes. The search metrics are connected to things that happen after the search, such as time to find an answer, support tickets, or errors caught. You can decide whether a metric movement matters to the people using search. The trap is attributing outcomes to search that have other causes. Tickets and time to find move for reasons that have nothing to do with relevance, so the link has to be examined, not assumed.
Rung 9: A/B tests on live traffic. Configurations are tested against each other on real users. You can decide what actually happens when real people use the change. The trap is running a test that cannot answer the question: too little traffic, several changes at once, or a success measure that was never defined before the test began.
What rungs 2 and 3 mean for document Q&A and RAG
The ladder applies to systems that answer questions over documents, with two adjustments at the first rungs that matter.
At rung 2, the log has to hold more than a query and a result list. For document Q&A and RAG, rung 2 means logging each question with the retrieved passages (their IDs and ranks), the answer, and its citations. Without the retrieved passages you cannot tell whether a wrong answer came from retrieval or from the model.
At rung 3, the labels can cover more than retrieval relevance. They can cover whether the retrieved passages were relevant, whether each citation actually supports the claim it is attached to, and, for scanned documents, whether the text was extracted faithfully in the first place. Accuracy can be lost before retrieval starts, and only a label on that layer will show it.
This is not a theoretical concern. A study of commercial legal research tools built on retrieval-augmented generation found that the tested tools "each hallucinate between 17% and 33% of the time" (Magesh et al., 2025). The point is not about any one product. It is that a system with retrieval in front of a language model can still be wrong in ways its users cannot see, which is the same invisibility the first section described.
How to climb one rung
Pick the rung you are on and move up exactly one. A log is the usual first step, because every rung above it depends on one.
On Liferay DXP, the free plugin records each search together with the results shown, which is rung 2. It is open source and sends nothing anywhere. From there, labelling a sample is rung 3.
On any other stack, you can build the same log yourself: each query, the results shown with their IDs and ranks, and a timestamp. For the statistics at rungs 3 to 7, the open-source evaluation harness is there for teams that want to run the calculations themselves.
Two habits make the climb safer. Write down the decision you want to make before you measure, so the numbers cannot be bent to fit it afterwards. And label with the people who know the documents, because a label is only as good as the judgment behind it.
The ladder is a framing
To repeat the caveat: this is TensorOpt's framing, not an industry standard. Other people draw the steps differently, and some of the steps could be split or merged. What the ladder is for is narrow. It makes the order of the work visible, and it makes one claim: a rung is only worth saying you have reached if you can name the decision it lets you make.
Laszlo Csontos
Author of Designing Hybrid Search Systems (Leanpub, 2026). Practitioner background in production hybrid search, embeddings, cross-encoder reranking, and retrieval evaluation.
Measure the search you can't see
Learn how to tell whether your search works, and how to measure a change before you make it. Five short emails over two weeks:
- Why misses stay invisible: users judge what is on the screen, never what is missing.
- The measurement ladder: nine rungs, and where your system sits.
- The first rung to climb: a log of each query and the results it returned.
- Getting started safely: the free plugin through security review on Liferay DXP, or, on other stacks, where accuracy is lost first.
- Two averages are not a decision: what to measure before you switch on semantic search or change a parser, embedding or model.
Then at most two short follow-ups. You also get practical files that are not on the site: a plugin rollout checklist (Liferay DXP) or a logging schema (other stacks), and a labelling kit.
Related Posts
A reranker moves NDCG@10 from 0.41 to 0.43 and the dashboard turns green. None of the major LLM-era eval platforms reports whether that 0.02 is signal or noise. The IR libraries settled this two decades ago.
June 25, 2026
A single NDCG number looks like a verdict but is often a coin flip. The variance across query subsets routinely exceeds the difference between two systems, and the IR community solved this in 1997. Most LLM-era eval tooling has not inherited the fix.
May 26, 2026
A reranker change moves NDCG@10 by +0.015 and the dashboard goes green. The paired bootstrap returns a 95% CI of [-0.003, +0.033], crossing zero. The point estimate said ship; the evidence says wait.
May 26, 2026