The architecture of image search

Search a database of documents and you are searching the thing itself. Search a database of images and you are searching a proxy: a text you manufactured, an embedding you computed, a hash you derived. The picture is never consulted. Everything you can retrieve was decided upstream, at index time, by code most search teams never look at twice.

That single fact explains almost every bad image search anyone has ever used. It is why a library of 200,000 photographs answers sunset with eleven results, why searching a SKU returns visually similar sneakers instead of the sneaker, why galaxy a52 case finds twelve assets when the shelf holds a hundred and seventy-eight, and why the demo always looks great. Relevance is the visible part. The hard part is upstream and invisible.

This article is the long version of how freedam's search actually works: the retrieval architecture, the parameters, the failure modes we hit in production, and the measurements that settled each argument. It is written for people who build search, not for people who buy it. If you want the product overview instead, that lives on the search page.

Part 1: Why images are the hardest thing in your database

A picture has no words in it

A row in an orders table has a customer name. A support ticket has a body. A PDF has a text layer. A JPEG has three colour channels and some EXIF. Nothing in it is a term, so nothing in it can be matched by a term. Every image search system in existence therefore does the same thing first: it builds a bridge from language to pixels. There are only four bridges, and each one is broken in its own way.

The temptation is to pick a favourite. Nearly every "AI search" retrofit picks the fourth bridge, and nearly every legacy DAM picks the second. Both choices are defensible in a demo and wrong in production, because real query logs are not homogeneous. Over a working library, queries fall into roughly four populations that want four different retrieval strategies:

• Identifier queries: 4471-RD, SS26-HERO-03, a GAID, an invoice number. These want exact lexical matching, and semantic similarity actively harms them: the nearest neighbours of a SKU embedding are other SKUs.
• Nomenclature queries: galaxy a52 case, taper plate, biodegradable mailer. These want lexical matching over a controlled vocabulary, plus tolerance for how the term was written down.
• Description queries: team celebrating outdoors, something calm for the newsletter header. Nobody tagged the asset with those words. Only an embedding can answer.
• Example queries: "more like this one", "did we already licence this photograph?". No text is involved at either end.

A single-lane system serves one of these well, one adequately, and two badly. So the interesting engineering question is not which retrieval method but how you combine them without the weak lane poisoning the strong one, and how you know, afterwards, that the combination did not silently drop half the answers.

The four failures everyone rediscovers

Before the architecture, the four dead ends, because each one leaves a fingerprint on the design that follows.

Pure keyword search fails on absence. Full-text search over "whatever text the asset happens to have" is only as complete as your metadata programme, and metadata programmes are always behind. It also fails on phrasing: the photographer wrote autumn, the marketer searches fall.

Pure vector search fails on precision. Embeddings are wonderful at "I'll know it when I see it" and structurally incapable of exactness. A vector index will always return k neighbours; it has no concept of "no match", and its confidence looks identical whether it found your asset or the nearest thing to it. Worse, on single-word queries the neighbourhood of a common noun is enormous, so a pure-vector system fills its result page for every query and looks like it is working.

Pure tagging fails on economics and on drift. Human tagging at scale is a permanent cost centre, and even when it is funded, the controlled vocabulary diverges from how people search within about a year.

Pure LLM search fails on physics. Asking a model to rank 200,000 assets per query costs seconds and dollars per search, and gives a different answer to the same query twice. Language models are excellent at interpreting a query and poor at scanning a corpus. They belong at the front of the pipeline, not in the middle of it.

The architecture below is what is left after taking all four failures seriously: manufacture as much text as possible, index it two different ways, fuse the two rankings with a method that does not require their scores to be comparable, then guarantee recall for the cases where the fusion's caps would otherwise lie to you, and report honestly when you cannot.

Part 2: Manufacturing the text

Everything downstream is decided here, and this is the layer that gets the least attention in most systems. freedam builds, per asset and per language, a single normalised text document, stored in asset_search_documents. It is not a view over the asset; it is a compiled artefact, rebuilt when the asset changes.

Three decisions in that picture are worth arguing about.

Field weighting is implemented as repetition. A field with weight n is appended n times to the document. This is crude and it is also the only weighting scheme that survives contact with an external BM25 index that has no notion of fields. It has a cost we measured the hard way: repetition inflates document length, and BM25's length normalisation (b = 0.75) penalises long documents. In one investigation, eleven product videos out-ranked the product images everyone expected for a place-name query. Not because of tokenisation, and not because of missing metadata, but because the videos' documents were 155 to 493 characters while the images' were 746 to 1,280. Short documents win ties. If you weight by repetition, you are also, quietly, ranking by verbosity. Know that before you tune anything else.

Vocabulary hierarchy is flattened into the document, with decreasing weight. An asset tagged with the leaf term espresso machine also gets small appliances (parent, weight 2) and kitchen (ancestor, weight 3) written into its document. This is what makes a taxonomy searchable rather than merely filterable: a query for the category finds the leaf without a join, and without the searcher knowing your tree.

The tsvector is a stored generated column, not a computed expression. Hybrid retrieval re-checks every candidate's term coverage after the ranked scan. Doing that with to_tsvector(content) per row means detoasting and re-parsing every candidate document on every search. As a GENERATED ALWAYS AS (…) STORED column it becomes a cheap column comparison. The trade-off is that the expression, including the language-to-stemmer mapping, is frozen at DDL time, so extending the stemmer map from 7 configurations to 20 required a migration that rebuilt the column and the affected BM25 indexes together. Stemming has to agree in three places at once, namely the stored tsvector, the BM25 index and the query-time plainto_tsquery, or coverage verification silently rejects valid matches.

The tokenizer is where recall actually dies

Here is the single most instructive bug we have shipped in search, and the reason this article exists.

A retail library used device-compatibility labels of the form Galaxy A52/A52S and iPhone 12/12 Pro. The labels were correct. They were on the assets. They were in the search document: a LIKE '%A52%' found 178 of them. And galaxy a52 case returned twelve results.

PostgreSQL's english parser classifies a slash-joined value as a single token of type file. ts_debug('english', 'Galaxy A52/A52S') yields the lexeme A52/A52S, not a52 plus a52s. The text was indexed. The term never existed.

Four things about this bug generalise to any image search you build.

A re-index cannot fix a tokenisation defect. The builder already emitted the text; regenerating produces the same lexemes. The fix has to change what the parser sees: here, emitting the split segments alongside the original so exact-form search keeps working.

The fix must be scoped, or it becomes the next bug. The first version applied the split to the whole normalised document and cheerfully shredded EXIF (1/100, 1:1.8/26), voltages (50/60Hz), URLs, EU directive references (2009/125/EC, in about twenty languages) and conjunctions (and/or, und/oder). Splitting is now applied only where vocabulary labels enter the document. Nobody searches for 2009.

Your instrumentation can be lying in the same direction. The token dictionary that powers typo tolerance and prefix matching is built by a different tokenizer, one that does split on slashes. So the dictionary confidently reported a52 present in 178 documents while BM25 could match twelve. A term-frequency table is not evidence that retrieval can find a term. Only retrieval is.

Measure both directions after a tokenisation change. The narrowing above cost exactly one test case, whose result set was saturated at the candidate cap and had been benefiting from its competitors' documents getting longer. A relevance panel that only measures the queries you were fixing will approve every regression you ship.

Part 3: Two retrieval lanes over one table

Lane one: BM25, inside the database

freedam runs BM25 through pg_textsearch on the same PostgreSQL instance that holds the assets, with no separate search cluster and no sync pipeline. That is a deliberate architectural choice with real consequences, good and bad.

What it buys: the search reads the same transaction as everything else, so a freshly uploaded asset is findable without waiting for a sync worker; permissions, embargoes and rights constraints are ordinary SQL predicates on the same query rather than a second authorisation model in a second system; and there is no class of bug where the index and the truth disagree. For a self-hosted deployment it also removes an entire stateful service from the operator's plate.

What it costs: you inherit PostgreSQL's planner, and you have to work with it rather than against it. Three of the things we learned:

• Indexes are per-language and partial. One BM25 index per language, built WITH (text_config=…, k1=1.2, b=0.75) WHERE language_code = 'xx'. Falling back to another language means falling back the index name, the language predicate and the text configuration together. An earlier version fell back only the index name, producing a partial index that could not serve the requested language's rows at all.
• The ordered index scan only stays fast while it has a row bound. An ordered BM25 scan takes its top-k path when the executor hands it a bound. Bounds survive a single-referenced CTE (the planner inlines it) and a derived table carrying its own LIMIT; they die in scalar subqueries and in CTEs referenced twice. The measured penalty for referencing one CTE a second time was 6 ms → 7.3 s. This is why membership flags ride up through a union projection with BOOL_OR instead of joining back to the CTE that produced them.
• Non-indexable predicates go after the scan, never inside it. Term coverage, meaning "does this document actually contain every query term?", is verified against the stored content_tsv on the bounded candidate set the scan returned. Put that predicate inside the ordered scan and the planner abandons the index for a merge join with per-row rescoring; we measured 7.4 s for one such query shape.

Two smaller details that matter in practice: the raw scan over-fetches at 3× the configured candidate count because pg_textsearch's ordered scan emits roughly three duplicate rows per document, and the whole thing is deliberately kept below a latency cliff: at the default bm25_top_k = 1000, the raw limit is 3,000 and the scan costs about 10 ms; raising the raw limit toward 6,000 falls off a cliff.

Lane two: embeddings, and why the threshold moves

Text and images are embedded into the same 768-dimensional space using an aligned pair of models, nomic-embed-text-v1.5 for language and nomic-embed-vision-v1.5 for pixels, served by a local sidecar, so a text query can be compared directly against an image vector, and no asset content is sent to a third party. Nomic's models are task-prefixed: documents are embedded as search_document, queries as search_query. Using the wrong prefix silently degrades everything, and nothing in the results will look broken.

Indexing is HNSW over cosine distance, m = 16, ef_construction = 200, as a partial index over non-null vectors. Query embeddings are cached briefly, including failure markers, so an embedding-service outage degrades to lexical-only search in milliseconds instead of hammering a dead sidecar once per keystroke.

The parameter that surprises people is the distance threshold, which is a function of query length, and not monotonic:

Query shape	Max cosine distance
1 word	0.35
2 words	0.42
3 to 5 words	0.45
6 to 10 words (or > 30 chars)	0.48
> 10 words (or > 60 chars)	0.40

Both ends are strict, for opposite reasons. A single word has a huge semantic neighbourhood: at a loose threshold, chair matches every interior photograph in the library, fills the result page, and looks authoritative while being useless. A very long query (someone pasting a full product title) produces a generic embedding that sits weakly near everything; loosening the threshold there returns the entire corpus in confidence order. The middle band, where the query is specific enough to have a meaning but short enough to keep it, is where semantic search earns its keep and where the threshold is loosest.

Single-word queries carry one further rule, which we consider the most important precision decision in the system: a single-word query requires a lexical hit. A row that matched only by vector similarity is discarded outright when the query is one word. Without that, niche single-term searches fill their entire result page with semantically adjacent noise, and the user cannot tell the difference between "we found 1,000 things" and "we found nothing and rounded up".

Part 4: Combining two rankings whose scores mean nothing to each other

A BM25 score and a cosine distance are not commensurable. Normalising them against each other is a popular idea that fails the moment a corpus, a language or a threshold changes, because the normalisation constants are properties of that result set. So freedam does not fuse scores. It fuses ranks, with Reciprocal Rank Fusion:

rrf_score(d) = w_bm25 / (k + rank_bm25(d))  +  w_vector / (k + rank_vector(d))
                                                          k = 60,  w = 1.0

Three implementation notes that cost us real incidents.

Cast your weights. PHP renders 1.0 as 1 when interpolated into a SQL string, and the rank columns are bigint, so weight / (k + rank) becomes PostgreSQL integer division and every score in the system is exactly 0. Every interpolated numeric that appears in a division is now wrapped in CAST(… AS FLOAT), with regression tests pinning both the hybrid and the BM25-only path. The failure is total and completely silent; the results still come back, just in an arbitrary order.

The floor is adaptive, and it has structural side effects you should know about. Low-scoring rows are cut by a minimum RRF score that depends on query word count: 0.008 for one word, 0.015 for two, 0.025 for three to five, 0.028 for six to ten, 0.012 beyond, each scaled by (w_bm25 + w_vector)/2 × 61/(k+1) so retuning k or the weights does not invalidate the constants. Two consequences follow arithmetically, and we would rather state them than have someone discover them:

• A two-word query can never return zero results. A vector-only row scores 1/(60 + rank), and the two-word floor of 0.015 admits exactly ranks 1 through 6. This is the origin of the "nonsense two-word queries return exactly six things" behaviour.
• At three to five words the floor of 0.025 exceeds the best possible vector-only score of 1/61 ≈ 0.0164, which makes a purely semantic match structurally impossible in that band. Every result in a four-word query carries a lexical rank.

Rows that provably match are exempt from the floor. When the query is short enough for term-coverage verification to run (ten words or fewer), every BM25 candidate is known to contain all the query terms. Those rows are genuine matches, so the floor is not applied to them at all; it only guards rows that reached the result set without a verified lexical match.

Part 5: The caps, and the recall they quietly eat

Every candidate-retrieval system has caps, and every capped system lies unless it is built not to. The defaults: 1,000 BM25 candidates (raw scan 3,000), 200 vector candidates, 1,000 rows in the final fused set.

A cap is invisible in the worst possible way. The paginator still reports a total. The page still fills. The assets beyond the cut simply do not exist, and nothing in the response says so. We have a name for the specific pathology, cap crowding, and a real example: a query with 1,953 documents containing every query term, competing for 1,000 candidate slots. The assets that lost were not less relevant; they were alphabetically unlucky in a tie-break.

The mechanism that fixes this is the part of the design we are proudest of, and it is simple once you see it: a guaranteed band rides into the candidate set at offset positions beyond every real scan position, and every cap in the pipeline is taught to admit band rows unconditionally.

Scope-aware retrieval: searching inside something

The most common form of cap-induced disappearance is scoped search. You are inside a collection of 400 assets, you search a term that 30 of them carry, and you get four, because the tenant-wide top-1,000 candidate pool was dominated by assets outside your collection and the collection filter was applied after retrieval. This is the single most reported "search is broken" complaint in every DAM, and it is not a relevance problem at all.

freedam resolves a scope for every search that carries a text term, and gives it its own retrieval band, unconditionally; there is no setting to enable. What the band does depends on how big the scope is, because the right algorithm changes with size:

An unscoped search pays none of this. The band is additional SQL that only materialises when a scope exists and is small enough to be worth one.

Part 6: Completeness as a first-class result

Here is the question almost no search system can answer: did I show you everything?

It matters far beyond user comfort. "Select all 1,240 results and apply this rights policy" is only safe if the result set is provably the whole set. A search that quietly truncated at 1,000 turns a bulk operation into a data-integrity incident that nobody notices for months.

So every freedam search returns a completeness verdict, computed by probes: extra COUNT columns that measure how many rows each candidate source actually produced, against the cap that source was operating under.

The rules around that verdict are deliberately conservative. A source may report Complete only when it provably holds every match. Per-member scoring is complete by construction; a deep scan is complete only if the in-scope match population stayed under the band cap and the scan never hit its row bound. A new candidate source defaults to Unknown. And a result delivered by a fallback, such as a typo correction or a prefix expansion, never inherits a verdict earned by the original term.

There is a broader discipline here that took us four separate incidents in a single day to internalise: a check whose failure mode is silence is not a check. A monitor that reported nothing while the job it watched had died. A tie-break test that stayed green after the tie-break was stripped from three SQL layers. Quality-suite notes that could not move an exit code. The question that finds them is always the same: if the thing this check watches broke right now, what would it print? If the answer is "the same as when it is fine", you do not have a check. You have a comment with a test-runner icon next to it.

Part 7: The other ways to ask

Text is one input. Three more matter for images specifically.

Typos and prefixes, in that order

The document text is also indexed with a GIN trigram index, which powers both "did you mean" and a typo fallback. Corrections are made per token against the corpus's own vocabulary, with a similarity threshold that scales with token length: short tokens demand a much closer match, because at three characters everything is similar to everything. Prefix expansion runs as a separate fallback with its own candidate budget. The ordering is not arbitrary: correcting a token that was actually a prefix of a real term produces confidently wrong results, so prefix behaviour is tried before a typo correction is allowed to rewrite the query.

Search by picture

Reverse image search ("have we seen this before?", "find the rest of this shoot", "is this a near-duplicate of something we already licensed?") takes no text at all, and needs a different index entirely: perceptual hashes of six tiles per image, looked up through multi-index hashing, then verified against a thumbnail vector and a vision embedding. Cheap, coarse and indexable to generate candidates; accurate and expensive to confirm them.

That subsystem has its own article, because the interesting part is where the textbook algorithm stops working on a real catalogue: finding the same image twice.

Faces

Faces get their own detector and their own 512-dimensional embedding space (InsightFace's buffalo_l), indexed with HNSW under cosine distance, in a sidecar that never sends an image off your infrastructure. The design constraint is not technical: a face has no name until a human gives it one. Identity is a human assertion, machine-propagated: the machine groups, the person confirms, and the confirmed name is then written into the searchable document so photos of <person> at the launch event works as an ordinary text query. See face recognition search.

Natural language, compiled to a filter tree

Conversational search is the fourth input, and the design decision that matters is where the model sits. freedam's assistant does not rank assets. It compiles your sentence into a filter tree, the same rule structure the advanced filter UI produces, with an explicit intent (replace, refine, add, remove, reset) that determines how it merges with the filters already active. That tree is then executed by the same compiler, under the same permissions, with the same completeness accounting as everything above.

The benefits are all consequences of that one choice: the interpretation is inspectable and editable (you can see the filters it chose and correct one), it is deterministic once compiled, it costs one model call rather than one per candidate, and it cannot invent an asset. The same surface is exposed to external agents over MCP, so an assistant driving your library gets exactly the retrieval semantics described in this article, and exactly the permissions of the token it authenticated with.

Part 8: How you know any of it works

Every parameter quoted in this article is a number someone argued about. The only thing that ends those arguments is a measurement harness, so freedam has one: a manifest of realistic cases with resolved ground truth, run through the production search path against a replica of a real library, reporting hard pass/fail plus recall@10, recall@50 and mean reciprocal rank per category. It is read-only by construction: it performs searches and ground-truth selects, and cannot write a setting or rebuild an index.

What that harness has actually taught us, in the order the lessons arrived:

1. The first serious run found fifteen hard failures in sixty cases. In a system that everyone using it described as "pretty good". Perceived search quality has almost no resolution below the top three results.
2. Failures cluster by mechanism, not by symptom. Those fifteen sorted into a handful of causes: scope truncation, tokenisation, cap crowding, document-length effects. Fixing a cause moved several cases at once; fixing a case moved one and broke another.
3. The obvious diagnosis is often wrong. Five recall regressions were confidently attributed by analysis to a tokenisation change. Measured, that change moved four of them by exactly zero, with identical recall@10, recall@50 and reciprocal rank to fifteen decimal places. The real cause was document length. Analysis proposes; measurement decides.
4. Every fix has a bill. Narrowing the slash-splitting to vocabulary labels, unambiguously correct, cost one case, whose result set was cap-saturated and had been accidentally benefiting from the bug. The panel went from 48 passes to 47, and we shipped it anyway, with the trade-off written down. A relevance change with no measured cost usually means an unmeasured cost.

What we would tell anyone building this

The transferable version, stripped of our specifics:

• Spend your effort at index time. Retrieval quality is bounded by the document you compiled. No amount of ranking cleverness recovers a term the tokenizer never emitted.
• Fuse ranks, not scores. Reciprocal rank fusion needs no calibration and does not break when a corpus, a language or a threshold changes. k = 60 is a fine place to start and rarely the thing worth tuning.
• Make thresholds a function of query shape. One distance cutoff for one-word and ten-word queries is one cutoff wrong twice. And require a lexical hit for single-word queries unless you enjoy explaining semantic noise.
• Treat caps as a correctness problem, not a performance setting. If your system has a top-k, some user is silently missing results today. Either guarantee a band that survives the cap, or report the truncation. Ideally both.
• Make your system say "I don't know". A search that can distinguish complete, truncated and unknown can safely authorise bulk operations. One that always reports a total cannot.
• Verify with the same SQL you shipped. A probe hand-written to mirror the result query will drift from it, and you will trust it right up until it matters.
• Check your instrumentation with the same suspicion as your code. Ours reported a term as present in 178 documents that retrieval could reach in twelve.
• Keep a relevance panel and read both directions. The cases you were not trying to fix are the ones that tell you what you broke.

Where this lives in the product

Everything described here runs on every freedam search: the gallery, the API, the TypeScript SDK, the assistant, and any AI agent connected over MCP. It runs on one PostgreSQL instance with pgvector and pg_textsearch, which is also what makes it practical to self-host: there is no search cluster to operate alongside it, and no asset content leaves your infrastructure to be embedded.

If you want the same subject at product altitude, read search and discovery. If you want to see it against a real library rather than a diagram, try the demo. It is a working instance with a seeded media library, and the interesting thing to do is search for something nobody would have tagged.