Measuring a search nobody thought was broken

Search is the only part of a product that can be badly broken while every test passes, every page loads and every user says it is fine. A search that has quietly lost two thirds of its matches returns ten plausible results in ninety milliseconds, exactly like a search that has lost none, and nobody looking at it can tell the difference, because they cannot see what is missing.

This is the fourth article in a series on how search works inside freedam. The first covered retrieval: manufacturing a searchable document, fusing BM25 with pgvector, guaranteeing recall under caps. The second took text out of the problem and looked at perceptual hashing. The third put it back and asked which language the index is in. All three quote numbers. This one is about where those numbers came from.

The short version: a sixty-case relevance panel, run manually against a replica of a real customer library of 22,944 assets. The first serious run found fifteen hard failures in a search everyone described as pretty good. Over the next eight runs it reversed a confident diagnosis, priced a fix nobody wanted to pay for, and caught itself being structurally blind to the feature it was validating.

Part 1: "Pretty good" is not a measurement

Nobody had reported a search problem. The library worked. People found things. The internal verdict, if you asked, was that search was pretty good.

The first serious run of the panel returned exit code 1: forty-five passes, fifteen hard failures, zero errored cases, across sixty cases on a 22,944-asset replica of a real retail catalogue.

The reason this can happen is not subtle, and it applies to every search product on earth.

Perceived search quality has almost no resolution below the top three results. A user types a term, glances at the first two or three, and forms a judgement. Whether the other 178 matching assets were reachable at all is invisible: no interface says "and 166 more that we cannot show you", and nobody scrolls to position 900 to check that position 900 exists. So a system can lose most of a result set and still feel, to everyone who uses it, pretty good.

The panel found galaxy a52 case returning twelve results when 178 assets carried that exact device. That number was there to be found on any day of the previous year. It took writing a case down.

Part 2: What has to be in a case before it is worth running

A relevance case is not a test. It is a claim about the corpus plus a claim about what the system should do with it, and the second is worthless if the first goes stale.

Each case carries an id, a category, the query as a retailer would type it, ground truth, and assertions split into hard and soft. Hard assertions fail the run; soft targets are reported and never fail anything, which sounds like a weakness and is a specific trap Part 8 comes back to. The interesting field is ground truth, and specifically the fact that it is almost never a list of asset ids.

A panel that dies when the corpus is replaced stops being run. That is the whole argument for derived ground truth, and it is practical rather than purist. The replica gets re-snapshotted from production whenever somebody needs fresh data, and asset ids do not survive that. A case pinned to 178 identifiers becomes an errored case the morning after a re-snapshot, and a suite that errors on a third of its cases is ignored within a week and deleted within a month.

So a derived case declares what it means rather than what it matched last time. Four specification types cover the panel: a metadata field equal to a value, one matching a prefix or substring, membership of a named collection, and a token-level title match. The resolver runs the specification against the database at the start of each run and hands the evaluator a fresh id set.

The schema also allows curated ground truth, an explicit id list, for cases where relevance is genuinely a judgement rather than a property. The panel currently uses zero of them. That is not an oversight. Curated cases are exactly the ones a re-snapshot invalidates, so every one of them is a future maintenance event, and the discipline of expressing a case as a specification usually reveals that the judgement was hiding an unstated rule.

One resolver detail is worth copying. A collection name has no unique constraint and a metadata key is unique only per asset class, so both lookups can genuinely match several rows. Picking the first silently would let a find_all case pass while half its assets were missing, because the expected set had quietly narrowed. Both refuse ambiguity instead: errored case, exit 2. A ground-truth resolver that guesses is worse than one that fails, because a case that resolves to the wrong set is a green light with no basis.

Twelve cases carry no id ground truth at all, because they measure something other than membership: the did-you-mean hint, the suggestions dropdown, the autocomplete strings, and the negative cases that must return nothing. And because "helpful" is a judgement call, every assist case carries a written rationale, frozen only after the live dropdown for that query was captured from the replica and reviewed by a person. An assist expectation invented at a desk measures the author's imagination.

Part 3: An instrument has to state its own conditions

Two things decide whether a relevance number means anything, and both are properties of the instrument rather than of the search: what configuration produced it, and what the candidate cap did to it before anybody looked.

Every run prints a settings snapshot before the first case executes: hybrid.bm25_top_k, hybrid.limit, hybrid.vector_top_k, hybrid.rrf_k, both fusion weights, typo.enabled and prefix.enabled. A misspelling result is meaningless without knowing whether typo correction was on. So is a recall figure without the candidate limit that produced it. Two runs whose numbers disagree are evidence of either a code change or a settings change, and the only way to tell is to have recorded both. Every claim in this article is qualified by a snapshot, which is why they can be compared at all: runs 1 through 8 all measured hybrid.limit 1000, hybrid.rrf_k 60, weights 1.0 and 1.0, typo.enabled true.

The snapshot also caught something nobody was looking for. On the tenant used for the first three runs, prefix.enabled printed as null. Not false. Null, meaning no settings row existed at all. The code default is true, so prefix matching was running and every result reflected it, but nobody had ever decided that, and reading the settings screen would never have told you, because a screen renders the effective value. The report printed the stored one. An effective value tells you what happened; a stored value tells you whether anybody decided it.

The second condition changes what an assertion is allowed to say. A total equal to the cap is not a count. On this tenant the candidate limit is 1000, so any query with more than a thousand plausible matches returns exactly 1000, which is the shape of the cap rather than a measurement of the corpus. A case whose ground truth is 2,189 assets cannot assert that all 2,189 appear in a result set structurally incapable of holding more than a thousand. Assert it anyway and the case is permanently red, and a permanently red case is worse than none: it trains everybody to read the run summary as "the usual failures plus whatever is new", which is how a real regression hides.

The second half of that figure is the load-bearing half. It would have been easy, and wrong, to treat saturation as a get-out: the result was truncated, so of course some assets are missing, so do not fail. That would make the whole class of cap defects invisible to the instrument built to find them. The settled product decision is that an asset whose title contains the query must always be findable regardless of rank or cap, so cap pressure is not an excuse for losing a directly matching asset. The report says so in the failure text: not findable within candidate window is a failure with its cause attached.

One sampling detail is worth copying. The sample is drawn by hashing rather than by a seeded random draw, so the same case draws the same fifty every run, and sampled recall 0.76 in run 1 against 0.76 in run 2 is a real comparison rather than two draws from the same urn.

Part 4: Failures cluster by mechanism, not by symptom

Fifteen failures spread across seven of the panel's fourteen categories looks like fifteen problems. It was six.

The finding in the first row is worth the detail, because it is the kind of defect that survives every reasonable investigation.

Device compatibility values in this catalogue look like Galaxy A52/A52S and iPhone 12/12 Pro. PostgreSQL's English parser classifies a slash-joined value as a single token of type file, so Galaxy A52/A52S is indexed as the lexeme a52/a52s and a search for a52 cannot match it. The text was in the document; a substring search found it; the tsvector did not contain it. Across the corpus, 178 documents held the raw text A52 and twelve held the lexeme, those twelve arriving via space-separated titles.

Every wrong diagnosis is available here. It looks like missing metadata, and the metadata is present. It looks like a stale index, and re-indexing regenerates exactly the same lexemes, so it fixes nothing while producing a plausible "we tried that". The correct fix is at index time: when a vocabulary label contains a slash, emit the split segments alongside the original.

There was also a booby trap. The tokeniser feeding the typo dictionary splits on slashes; the document builder did not. So the dictionary confidently reported a52 in 178 documents while BM25 could reach twelve. The instrumentation was reporting the answer the fix was supposed to produce, before the fix existed.

Run 2 measured the tokenisation fix and the scope change separately, in one four-point design. galaxy a52 case went from 12 results to 178 with zero assets unfindable, iphone 12 case from sampled recall 0.06 to 1.00, and the two scoped cases from 1 of 28 and 10 of 24 to full membership. Between the scope-off and scope-on arms, nothing else in the panel moved at all: not a verdict, not a total, not a recall figure, not an MRR. The design constraint that unscoped searches pay zero additional cost stopped being an assertion and became a measurement.

Part 5: Analysis proposes, measurement decides

This is the part of the story worth the price of the whole instrument.

The tokenisation fix had a side effect. It first shipped applying its rule to the entire normalised document rather than to vocabulary labels only, so it split every slash-joined token from any source: EXIF shutter speeds like 1/100, voltages, URLs, EU directive references like 2009/125/EC, ordinary conjunctions. Roughly half the corpus grew, and documents carrying a slash averaged 3,758 characters against 917 for those that did not.

BM25 normalises by document length, so every document that grew lost a little score on every term it contained and every document that did not grow gained rank for free. Sure enough, five cases regressed on recall@10 between run 1 and run 2, two of them from a perfect score to zero. The analysis wrote itself: the expansion inflated document lengths, inflated lengths demoted the ground truth, and narrowing the expansion to vocabulary labels would reverse it while keeping the recall recovery.

That was the recorded prediction. Run 3 narrowed the expansion, rebuilt the corpus, and measured it.

The narrowing did exactly what it was designed to do at the corpus level. The lexeme a52 still matched 178 documents; the lexeme 2009, which had been leaking out of 2009/125/EC, matched zero. And four of the five predicted reversals did not move by a hair.

The real mechanism was visible once someone stopped looking at tokens and started looking at lengths. The query greenland was being won by eleven Product Video documents of 155 to 493 characters, each essentially the title repeated by its field weight plus a little codec metadata. The first product image, which is what the case's ground truth consists of, arrived at rank 12 at 751 characters against a ground-truth average of 1,280. A three-to-eight-times length ratio decides an ordering on its own, and the document at the boundary contained no slash at all, so neither version of the expansion had ever touched it.

A wrong diagnosis that is coherent, specific and mechanism-level is much more dangerous than a vague one, because it survives review. Four people can agree with it. The only thing that disagreed was a measurement, and the only reason a measurement existed is that somebody had written down the prediction beforehand in a form that could be scored.

The narrowing shipped regardless, because it was still the right change. It also cost a case. compat-iphonexxs-01, the query iphone x case, went from sampled recall 1.00 to 0.74 and flipped from pass to hard failure, because its result set saturates at the cap in every run and the over-broad expansion had happened to lengthen its competitors just enough to admit more iPhone X and Xs assets. Take the pollution away, the competitors get shorter, score better, and crowd back in. The panel went from 48 passes to 47, and the trade-off went into the record alongside the fix.

A relevance change with no measured cost usually means an unmeasured cost. Ranking is close to zero-sum inside a capped window: something that rises pushes something down. If your instrument reports only improvements, either you found a genuinely free win, which happens, or your case set has nothing pointed at what you displaced, which happens more often. A case that fails for a known, priced reason is not a broken test; it is an accepted liability with a name. The failure mode to avoid is not "the suite is red", it is "nobody can say which reds are on purpose".

The habit worth stealing is smaller than the finding: write the prediction down, per case, before the run. Not "this should improve recall" but "these three cases should flip and these two should not move". Run 2 scored ten such predictions and got ten right, and the informative half was the negative ones, because iphone 16 case and airtag holder were the cases that would have exposed a misdiagnosis. Run 3 scored five and got one right. Both runs were worth the same, and only one of them felt good.

Part 6: A panel can only see what it has cases for

Run 4 was a canary for a different feature: the title-findability guarantee, which promises that an asset whose title contains the query is findable by that query regardless of rank or cap. Control arm off, treatment arm on, same corpus, same build, same settings. One caveat travels with every number from here on: run 4 measured a fresh replica of production, which has deliberately not been re-indexed since the tokenisation fix shipped, so its corpus is the pre-fix one. Both arms share it, which is all an A/B needs, but the run-4 verdicts are not comparable case by case with run 3 even where the totals coincide.

The result was flat on verdicts, 47 passes in both arms, and unambiguous on metrics. Mean recall@10 fell from 0.6667 to 0.6333, mean recall@50 from 0.7446 to 0.7338, and three cases regressed. Not one case improved on any relevance metric.

Read at face value, that is a feature which costs relevance and delivers nothing. The correct reading is different, and the run record says so in its own headline: not one of the sixty cases asked whether a beyond-cap titled asset is findable by its title, which is the entire contract the guarantee exists to satisfy. Every case graded retrieval against metadata-derived ground truth: a range, a colour, a device list, a GTIN. The instrument was structurally incapable of observing the benefit it was being used to judge.

Run 4 therefore decided nothing except that the panel needed cases. Run 5 added three.

The control case is the one that looks like padding. Its title band is 188 assets: below the 1,000 candidate cap, and below the 200-asset threshold at which membership assertions downgrade to sampling. Nothing is ever cut, so it passes with the guarantee off and with it on, identical in both arms.

A case set where every case only passes with the feature enabled proves that the cases were written to the answer. It cannot distinguish "the guarantee works" from "somebody chose three queries the guarantee happens to help". The control is the case that would have gone red if the implementation were doing something broader than advertised, such as reordering results that were already fully reachable. It stayed green and unchanged, which is a claim the other two cases cannot make on their own.

The addition also let the panel see something the sixty cases never could. title-product-01 returned 2,587 results against a ground truth of 2,156, meaning 431 non-title matches survived alongside a band more than twice the size of the cap. An earlier version of the guarantee had numbered the band and the ranked pool together, so the band took every position under the cap, got re-admitted anyway by its own clause, and silently deleted the ranked matches it displaced. Those 431 assets are that fix, visible on a real corpus.

Then the panel priced the cost, and paid for one more run that returned nothing. With the guarantee on, the whole title band still sorted ahead of every content-only match, because the final ordering puts a phrase-match boost first and every member of a single-word query's band earns that boost by construction. The first fix added a demotion inside the search plan's SQL, and run 6 came back byte-identical to run 5b: not one metric on one case moved. That plan is consumed as a joined subquery, and a subquery's ordering does not survive the join; three places emit the clause and only two govern what a user sees. Moving the demotion to both consumers took the panel to zero regressions on any relevance metric across all sixty-three cases, with both title cases still passing on full bands of 3,339 and 2,587. An unchanged result is different in kind from a partially changed one, and only one of the two means the fix did not work.

Part 7: Change is not correctness

Everything so far comes from one instrument with a fixed blind spot. Its cases probe failure modes we already suspected and its ground truth is derived from metadata, so it answers "is the right thing findable" precisely and cannot answer "would a real user notice" at all. Part 6 is what happens when you forget that.

The second instrument is built from the production query log, and it is worth being clear about the kind of evidence it produces before quoting any of it. It has no ground truth. It compares two arms of the same corpus and reports what moved. That measures change, not correctness: a query whose top ten is identical is unchanged, which is not the same as verified, and a query that gained results has gained reachable results, not good ones.

The source was analytics_searches on one production tenant: 140 rows covering 99 distinct terms over ten weeks. Only aggregate fields were extracted, the term, its occurrence count, the search type and the recorded result count. No user id and no session id left the table. A query log is one of the most personally revealing artefacts a product holds, and a relevance instrument does not need to know who typed anything.

Of the 99 distinct terms, 55 were keyword searches and 44 were serialised advanced filter trees. The 44 were excluded with a stated reason rather than quietly dropped: a tree with no full-text leaf never reaches the hybrid search path, so including them would have inflated the "nothing changed" denominator with queries that could not possibly change. An exclusion without a reason is a number you cannot audit. The 55 that remained look nothing like a relevance panel: SKUs, filenames with extensions, brand phrases, natural-language descriptions, case variants of the same word counted separately, one misspelling, one Danish term, and a single paste of 79 SKUs.

The result was strictly additive. The top ten was identical on all 55. Five totals grew, every one a query already pinned at the 1,000 candidate cap, so the gain was purely beyond-cap reachability: greenland 1000 to 1014, iphone 1000 to 1010, phone and blue products 1000 to 1005. Nothing lost results, and the two zero-result queries were the same two before and after.

That is not a result about quality. It is a result about blast radius, which is the thing a curated panel is worst at estimating, because its sixty-three queries were chosen precisely for being sensitive to the mechanisms under test while the replay's were chosen by customers for their own reasons. Both facts are true at once, and neither instrument can state both.

It did find one thing the panel could not. With the band merged in, the reported candidate limit became null on 52 of the 55 queries while the truncation flag stayed set on all of them, so the interface goes from "may be truncated, limit 1000" to "may be truncated, depth not describable by one number". That is honest, and it is a copy decision to make before rollout rather than discover afterwards.

Latency came with its own lesson in measurement hygiene. The first pass suggested the title band made search faster, which is not a thing a band can do: the arms had run sequentially and the second inherited warm caches. Warm against warm, the difference is about five milliseconds of mean, which is noise, with the worst individual queries eight to ten milliseconds slower. If your A and B did not run under the same cache conditions, you measured the cache.

The limit is the sample. Fifty-five distinct terms from one tenant over ten weeks, skewed towards SKU lookups, is a small window on exactly the query shape least likely to be affected by the thing being tested.

Part 8: Checks that cannot fail

Everything above assumes the instrument works. That assumption failed four times in a single day, in four places, in one shape.

A missing check is obvious because nobody wrote it. A check that cannot fail looks like coverage. It is in the diff, in the test list, and green on every run, and if the thing it watches broke right now it would emit exactly what it emits when everything is fine.

A computed exit code that nothing consumed. The runner carefully returned 0 for pass, 1 for hard failures and 2 for configuration errors. The trait that invoked it discarded the integer and reacted only to a thrown exception, and the runner was deliberately designed never to throw, so every real invocation exited 0 regardless of how many cases failed. The suite's own tests passed, because the test runtime takes an earlier branch that does propagate the return value. The only exercised path was the only correct one.

Metrics that report but cannot gate. Soft targets emit a note saying the first relevant result arrived at rank 17 against a target of 10, and notes do not move an exit code. After the document-builder change, five cases lost top-ten relevance, two falling from perfect to zero, and four of them still reported pass.

A monitor that reported silence while the job it watched died, because it filtered for progress markers only and a crash produces no marker. A tie-break test that stayed green after the tie-break was stripped out, because the assertion happened to hold for reasons unrelated to the behaviour it named.

They share a shape: a signal exists and no consumer can act on it. Nobody writes a branch that does nothing on failure deliberately; it arises where a value crosses a boundary and the far side was written to a different contract.

The question that finds them is one sentence long. If the thing this check watches broke right now, what would it output? If the answer is "the same as when it is fine", it is not a check. The detection method that works is more embarrassing and far more reliable than the question: break it on purpose, then confirm red. Of the four, the two caught by fault injection were caught quickly; the two caught by reading took far longer and only worked because somebody was already suspicious, which is not a technique.

The cost of the two that shipped is the argument for spending the hour. The exit-code defect would have made a full replica run report success unconditionally, so the entire measurement exercise in this article would have produced a green light with no basis. The soft-target defect is live: a merged change moved a search's first relevant result from rank 1 to rank 17 while the suite that exists to catch exactly that reported a pass. It is documented as diagnostic rather than gating, which is the minimum acceptable resolution. A number that is reported but gates nothing will be read as a guarantee unless you write down that it is not.

What we would tell anyone building this

The transferable version, stripped of our specifics.

• Assume you are wrong about your own search until an instrument disagrees. Perceived quality has no resolution below the top three results, so a system can lose most of a result set and still feel fine to everyone who uses it. Fifteen failures in sixty cases, and not one support ticket.
• Express ground truth as a specification, not a list of ids. A panel that dies when the corpus is replaced stops being run, and a suite nobody runs is deleted within the month. Curated cases are future maintenance events; keep them near zero.
• Make the report state the configuration it measured, and print stored values rather than effective ones. A recall figure without its candidate cap is a rumour, and the gap between stored and effective tells you whether anybody actually decided.
• Downgrade the assertion under a cap, never the verdict. Sample instead of asserting full membership, but keep a missing direct match a hard failure, or the instrument goes blind to exactly the defect class it was built for.
• Sort failures by mechanism before you fix any of them. Five failures in one category were three bugs needing three fixes, and only one was about retrieval limits. Fixing a cause moves several cases at once; fixing a case moves one and breaks another.
• Write the prediction down before the run, per case, including what should not move, and expect a bill. The negative predictions expose a misdiagnosis. Ranking is close to zero-sum inside a cap, so a change that shows only improvements usually means nothing in your case set points at what you displaced.
• Check whether your instrument can see the thing you are asking it about, and include a control that must pass with the feature off. Zero better and three worse was a true measurement and a false conclusion, because not one case exercised the contract under test. A case set where everything only passes with the feature on shows the cases were written to the answer.
• Run two instruments with different blind spots, and ask of every gate what it would output if the thing it watches broke. A curated panel knows the right answer and not what anyone asked; a query log knows what everyone asked and not the right answer. Then break your checks on purpose and confirm red, because reading finds these only if you were already suspicious.

Where this lives in the product

None of this is a feature. It is the instrument that decides which features ship, and its output is visible only in the behaviour of ordinary searches: the gallery, the REST API, the TypeScript SDK, and any AI agent connected over MCP. The fixes it arbitrated are the ones described in the first article: the recall guarantees under caps, the candidate selection, the completeness reporting that lets a truncated result say so. The same subject at product altitude is search and discovery, and the fact that all of it runs inside one PostgreSQL instance is what makes it reproducible on a replica at all, which is also what makes freedam practical to self-host.

The limits deserve stating as plainly as the results, because an instrument that oversells itself is the exact failure this article is about. The panel is manually run and deliberately not in continuous integration: it needs a replica of a real customer corpus, an embedding sidecar and about ninety seconds per run, and its numbers mean nothing against any other corpus. Its queries are English only, because the catalogue it was built for is English, so it measures none of the multilingual behaviour in the previous article. Its cases resolve their ground truth freshly on every run and survive a re-snapshot, but a changed corpus still changes the numbers: runs 4 through 8 measured a fresh replica of production, un-reindexed since the tokenisation fix, so they are not comparable case by case with run 3 even where the totals happen to match.

Next in this series: faces and person search, where the interesting problems stop being about retrieval quality and start being about who is allowed to assert that two faces are one person.