What it actually takes to build a good image search

In February 2000, Jennifer Lopez wore a green Versace dress to the Grammys, and for a few weeks afterwards it was the most searched thing on Google. Google could not show it to you. It could show you ten blue links, some of which led to pages that might contain a photograph. Eric Schmidt later wrote that it was the most popular query they had ever seen, and that it was the reason they built an image search at all. Google Images launched in July 2001, indexing about 250 million pictures.

I like this story because of the part people leave out. Google Images did not look at the pictures. It could not. It indexed the filename, the alt text, the caption, and the words that happened to sit near an <img> tag on the page, and it matched your query against those. The dress was findable because thousands of web pages had written "Jennifer Lopez" and "Versace" and "Grammys" next to it. The pixels were never consulted.

Twenty-five years, several generations of computer vision and one deep learning revolution later, this is still how image search works. Not mostly. Entirely. And I think it explains most of what goes wrong with it.

I run freedam, a digital asset manager, which is a fancy name for "the place a company keeps its 200,000 photographs and cannot find any of them". Search is the product. We have spent more engineering time on it than on anything else, and this is an attempt to write down what that taught us, for people who might build one, or who are wondering why the one they have is so bad. There is a much more technical version of the second half with the SQL and the parameters, if you want it. This one is about the shape of the problem.

A picture has no words in it

Here is the thing to hold onto. A row in an orders table contains a customer name. A support ticket contains a body. A PDF has a text layer. A JPEG has three colour channels and some EXIF, and nothing in it is a term, so nothing in it can be matched by a term.

So every image search ever built does the same thing first: it manufactures a stand-in for the picture that a computer can search, and then searches that. There have only ever been four kinds of stand-in, and the history of the field is really the history of people discovering each one, getting excited, and finding out where it breaks.

Words someone typed. A filename, a title, a caption, the text of the page around the image. This is what Google Images launched on, and what AltaVista's Photo Finder had launched on three years earlier. It is cheap and it is surprisingly good, because humans are good at naming things, and it fails completely the moment nobody typed anything. A folder called DSC_0001 to DSC_4817 is a black hole.

Words from a vocabulary. Tags, categories, a taxonomy. Flickr made tagging fashionable in 2004 and for a while "folksonomy" was a word people said in meetings. Every stock photo agency and every corporate library runs on this. It is precise and filterable and it is expensive, because someone has to do it, which means it is never finished, and the vocabulary drifts away from how people actually search within about a year.

Words a machine wrote. OCR on the text inside the image, a caption generated by a vision model, a transcript for a video. Luis von Ahn's ESP Game in 2004 had a wonderful version of this: two strangers shown the same picture, scoring points when they typed the same word, and the words they agreed on became labels. Google licensed it as Google Image Labeler in 2006 and ran it for five years. Today the labeller is a model and it never gets bored, which means this bridge is now plentiful, noisy, and blind to anything specific to your business. The model knows it is a sneaker. It does not know it is SKU 4471-RD.

No words at all. Describe the picture as numbers, and compare the numbers. IBM's QBIC did this in 1995, letting you search a database by sketching a colour layout; it was a research triumph and nobody searched that way. TinEye in 2008 and Google's search-by-image in 2011 made the reverse version mainstream: give me a picture, find copies of it. Then in 2021 CLIP put text and images into the same numerical space, and for the first time you could type a dog on a skateboard and find one that nobody had ever tagged. This is the bridge that every "AI search" demo of the last four years is built on.

Notice that nothing on this list replaced anything above it. Google Images today still leans on surrounding text. Your stock agency still pays taggers. Each new bridge got added to the pile, because each one answers questions the others cannot, and the mistake that almost everyone makes, including us at first, is to pick a favourite.

What the research was chasing in the meantime

It is worth spending a few minutes on what the academics were doing while the products above shipped, because the research followed one question for thirty years and then, quite suddenly, answered it, and the answer is much more limited than the demos suggest.

The phrase "content-based image retrieval" was coined in 1992, by Toshikazu Kato at Japan's Electrotechnical Laboratory, for the idea that you could search pictures by what was in them rather than by what someone had written about them. IBM's QBIC was the first serious system to try. It could match colour histograms, textures and shapes, and you queried it by sketching. AltaVista's Photo Finder in 1998, built with a company called Virage, actually shipped a "visually similar" button to the public, and the public, overwhelmingly, typed words instead.

The reason was named in 2000, in a survey paper by Smeulders and colleagues that every researcher in the field has cited since: the semantic gap. The gap between what a computer can extract from pixels (this region is greenish, this edge is vertical) and what a person means by a query (a dress, a celebrity, an awards ceremony). Colour histograms live on one side of that gap. Every query anyone actually types lives on the other. For the next twenty years the research programme was, essentially, "close the gap".

It closed in stages, and each stage ended up in a product. Local features (SIFT, 1999) and the "Video Google" trick of 2003, which treated visual patches as words and indexed them like text, are the direct ancestors of TinEye and Google's reverse image search: same-picture matching at web scale, which is a different and easier problem than meaning. ImageNet in 2009 made a million labelled images available, and AlexNet in 2012 showed that a neural network trained on them produced features that beat every hand-designed descriptor. Then in 2013 a paper from Google called DeViSE did something that now seems obvious and then seemed strange: it trained the image features to land near the word vectors for their labels, so that an image of a thing and the name of the thing sat close together in one space, and a model could recognise categories it had never been trained on. CLIP in 2021 was that idea scaled to 400 million image-text pairs scraped from the web. The gap that had been the field's central problem since 1992 was, for descriptive queries, mostly closed.

What interests me is the word descriptive. CLIP can find a dog on a skateboard. It cannot find SS26-HERO-03, and it has no idea that the photo is inside the collection you are currently looking at, and it will never tell you that it skipped 953 results. None of those were ever research problems. They were not interesting enough. They are, in my experience, most of what goes wrong in a real search, and the tools that handle them came from a different lineage entirely: BM25 from 1994, the keyword-scoring formula behind nearly every text search you have ever used, which nobody has meaningfully improved on since; reciprocal rank fusion, a 2009 paper of barely two pages; HNSW in 2016, which is what makes nearest-neighbour search over millions of vectors fast enough to put behind a search box. The third lane in the diagram is where a search engine is actually built, and the research that closed the semantic gap is one component in it.

What people actually expect

Before the architecture, I want to spend some time on the users, because the expectations are where most of the design constraints come from, and they are not what a search engineer would choose.

Everyone who uses an image search learned what one is from Google Images, and Google Images taught them three things. It is instant. It is never empty; there is always a grid, and it scrolls forever. And it is never wrong, exactly, because there is no right answer to sunset on the open web. You are browsing. If the third result is nice, the search worked.

A company's photo library is a different animal, and the Google mental model is actively harmful there, in ways that took me years to see.

In a library, the answer is usually specific. You are not looking for a sunset. You are looking for the sunset from the Lisbon shoot, the one with the bridge, because it was in last year's annual report and the designer needs it again. There is one correct result, or twelve, and you know roughly what it looks like. A grid of 400 plausible sunsets is not a success. It is a different kind of failure.

In a library, the query is very often not a description at all. I have read a lot of query logs. They are full of 4471-RD, SS26-HERO-03, invoice numbers, the model name of a phone case. Nobody types those into Google Images. Everyone types them into a DAM, and these queries are murder for anything semantic: the nearest neighbours of a SKU, in embedding space, are other SKUs.

In a library, people search inside things. Inside a folder, inside a collection, inside this campaign. "Find the hero shots in this shoot" is the single most common search there is, and, as I will get to, it is the one that most systems get wrong in a way nobody can see.

In a library, "no results" is a legitimate answer, and a valuable one. Has anyone already shot a photo of the new store in Madrid? If the honest answer is no, I want to hear no, so that I can commission one. A system that has absorbed the Google reflex and fills the grid with Barcelona and Seville is lying to me, and costing me money.

And in a library, people act on the whole result set. "Select all 1,240 results and apply this licence expiry." Nobody does that on Google. Everybody does it in a DAM, and it is only safe if 1,240 is actually all of them.

On top of all that, people judge a search by its top three results, and cannot see what it did not return. This is the property that makes bad image search so durable. A search that has silently lost two thirds of its matches still returns a full page of plausible things in ninety milliseconds, exactly like a search that has lost nothing, and nobody looking at it can tell the difference. We once had a library where everyone, us included, said search was "pretty good". When we finally measured it, fifteen of sixty realistic queries were hard failures. Nobody had reported a single one.

So, the actual brief for a good image search, at least for a library, reads something like this. Find the specific thing. Handle identifiers and nomenclature as well as descriptions. Work properly inside a folder. Say "no" when the answer is no. Never silently drop anything. And know, and be able to tell the user, whether it has shown them everything.

That last one is the one nobody ships. The rest of this article is about how we tried to.

Everything is decided before anyone searches

The first thing we got wrong, and I think most teams get wrong, is where to spend effort. Search quality feels like a ranking problem, because ranking is the part you can see. It is mostly an indexing problem. Whatever you did to the image at upload time is the ceiling on what any query can ever find, and no amount of cleverness at query time recovers a word that was never written down.

So freedam compiles, for every asset, a text document. Not a view over the metadata; a real, stored artefact, rebuilt whenever the asset changes, one per language. Everything goes in: title, filename, description, creator, copyright, location, every keyword, every AI-generated tag and caption, the OCR text, every custom metadata value, the names of every collection the asset sits in, the confirmed names of every person whose face is in it, the subtitles if it is a video. And the vocabulary with its ancestors: an asset tagged espresso machine also gets small appliances and kitchen written into its document, so that searching the category finds the thing without anyone needing to know our tree.

Two things about this document turned out to matter far more than I expected.

The first is weighting. A title should count for more than a line of OCR, and the only weighting scheme that survives contact with a keyword index that has no notion of fields is embarrassingly crude: you write the important field into the document several times. It works. It also has a side effect that cost us a real investigation. Keyword scoring normalises for document length, so short documents win ties, and if you weight by repetition you are also, quietly, ranking by verbosity. We once had eleven product videos outranking the product photographs for a place-name query, and the reason was not metadata or tokenisation. The videos had thin documents, 155 to 493 characters. The images had rich ones, 746 to 1,280. Being well-described was being punished.

The second is the tokenizer, the piece of code that decides where one word ends and the next begins, and I mention it mainly because of a blunder of ours that makes the point. A retail customer labelled accessories by device compatibility, Galaxy A52/A52S, and for months a search for galaxy a52 case found twelve of the 178 assets carrying that label. The text was in the index. But PostgreSQL's parser treats anything with a slash in it as a single token, like a file path, so the index contained A52/A52S and never contained a52. Nothing in the data looked wrong, no error was ever raised, and no amount of re-indexing would have helped, because the fix is upstream, in what the parser is shown. That is the general lesson, and it is the whole reason I keep insisting on index time: the search document is the ceiling, and the ceiling is set by code that most teams never read.

Two ways to search, one database

Once you have the document, there are two useful ways to search it, and they are good at opposite things.

The first is old-fashioned keyword search, and the algorithm behind it is called BM25. The name is unhelpful (it stands for "Best Match", and 25 is just the version number that stuck) but the idea is simple enough to say in a sentence. A document scores higher for a query word when that word appears in it more often, when the word is rare across the whole library, and when the document is short, so that a word in a ten-word title counts for more than the same word buried in two pages of OCR. That is the entire formula, it is from 1994, it is what powered every search engine of the early web and what still powers most of them, and nothing has managed to replace it for the thing it does well, which is exactness. It is the only thing that can find 4471-RD. Its weakness is absence and phrasing: if the photographer wrote autumn and the marketer types fall, BM25 shrugs.

The second is the CLIP-style approach: turn the document, and the image itself, into a point in a 768-dimensional space, turn the query into a point in the same space, and return whatever is nearby. This finds team celebrating outdoors when nobody tagged it. Its weakness is the opposite one. A vector index always returns k neighbours. It has no concept of "nothing matches". Its confidence looks identical whether it found your asset or merely the nearest thing to it, and that is precisely the Google-grid failure mode I described above, now with a neural network behind it.

We run both, and the thing I would argue hardest for is that we run both inside PostgreSQL, on the same instance that holds the assets, in the same row. BM25 through the pg_textsearch extension, vectors through pgvector with an HNSW index. There is no Elasticsearch, no Pinecone, no sync pipeline.

That is an unfashionable choice and I want to be honest about the trade. What it buys is enormous: a freshly uploaded asset is searchable in the same transaction that inserted it; permissions, embargoes and usage rights are ordinary WHERE clauses on the same query, not a second authorisation model in a second system; and there is no class of bug where the index and the truth disagree. For customers who self-host, it also removes an entire stateful service from their plate. What it costs is that you inherit the Postgres query planner, and you have to learn its moods. We have a query shape that runs in 6 ms and a nearly identical one, which references the same subquery twice, that runs in 7.3 seconds, because the second reference loses the row bound that lets the index do a top-k scan. Most of our search engineering is of this kind: not algorithms, but persuading a planner to do the thing you meant.

The embeddings come from a pair of models (nomic's text and vision models, trained to share a space) running in a sidecar we host ourselves, so no customer image ever leaves their infrastructure to be embedded. Two details I wish someone had told me. The models are task-prefixed: documents get embedded as search_document, queries as search_query, and if you get that wrong, nothing errors and everything silently gets worse. And we cache query embeddings including failures, so that when the embedding service falls over, search degrades to lexical-only in milliseconds rather than hammering a dead sidecar once per keystroke.

How close is close enough?

The vector lane needs a cutoff: beyond some distance, a neighbour is not a match. I assumed this was one number. It is a function of how many words you typed, and it is not even monotonic.

Query	Max cosine distance
1 word	0.35
2 words	0.42
3 to 5 words	0.45
6 to 10 words	0.48
more than 10 words	0.40

Both ends are strict, for opposite reasons. A single word like chair has a gigantic semantic neighbourhood: at a loose threshold it matches every interior photograph in the library, fills the page, and looks authoritative while being useless. A very long query, which in practice is someone pasting an entire product title, produces a vague, averaged-out embedding that sits weakly near everything; loosen the threshold there and you get the whole corpus back, in confidence order. The middle band, where the query is specific enough to mean something and short enough to keep meaning it, is where semantic search earns its keep.

Single words get one more rule, and I consider it the most important precision decision in the whole system: a one-word query must have a lexical hit. If the only reason a row showed up is that its vector was nearby, and the user typed one word, we throw it away. Without this, every niche single-term search fills its page with semantically adjacent noise, and the user cannot tell "we found a thousand things" from "we found nothing and rounded up". With it, madrid returns nothing when there is nothing, which is the answer I wanted.

Combining two rankings that do not speak the same language

A BM25 score and a cosine distance have nothing to do with each other. The popular idea is to normalise them onto a common scale and add them, and it works right up until the corpus, the language or a threshold changes, because the normalisation constants were properties of that result set.

So we never compare scores. We compare ranks. Each lane produces an ordered list; a document at position r in a list contributes 1 / (60 + r); a document absent from a list contributes nothing from it; sum the two. That is Reciprocal Rank Fusion, it is from a short 2009 paper, and it has the lovely property that a document ranked 40th lexically and 3rd semantically will beat one that is 2nd in one list and absent from the other. It needs no calibration and it does not break when you change the corpus.

The part where the search lies to you

Everything so far is about relevance, which is the visible part. The part that I think actually separates a good image search from a bad one is what happens at the edges, and the edge is the cap.

Every retrieval system has one. You cannot score 200,000 documents per keystroke, so you take the top 1,000 lexical candidates and the top 200 vector candidates and rank those. Sensible. And invisible in the worst possible way, because the paginator still reports a total, the page still fills, and the assets beyond the cut simply do not exist. Nothing in the response says so.

Here is what that does in practice. You are inside a collection of 400 assets. You search a word that 30 of them carry. You get four. Why? Because the top-1,000 candidates were chosen from the whole library, the filter "inside this collection" was applied afterwards, and the other 26 were in positions 1,001 and up. This is, by a wide margin, the most common "search is broken" complaint in every DAM I have ever looked at, and it is not a relevance problem at all. It is a plumbing problem. The results it does show are perfectly relevant. It has just lost most of them.

We have a second name for the general case, cap crowding: a query where 1,953 documents contain every search term, competing for 1,000 slots. The ones that lost were not less relevant. They were alphabetically unlucky in a tie-break.

The fix is not clever, and in hindsight the bug was an elementary misunderstanding of how candidate retrieval works; I suspect most systems that have it have it for the same reason. Some rows must be reachable regardless of where they ranked. Members of the collection you are searching inside. Assets whose title literally contains the term. So we run a second, guaranteed retrieval for those, and we put them into the candidate set at positions beyond every real scan position, and we teach every cap in the pipeline, the score floor, the final LIMIT, all of it, to admit guaranteed rows unconditionally. Because they sort after everything that was genuinely ranked, the first page the user sees is byte-for-byte the same as before. The band changes what is reachable, not what is first. That separation is what makes it safe to ship.

How you build the guaranteed band depends on how big the scope is, and the small case is my favourite, because it is not an optimisation. If you are searching inside a collection of forty assets, we do not scan anything. We score all forty, directly. That is complete by construction, at any library size, and it costs about half a millisecond per member. Up to a couple of thousand members we add a brute-force vector comparison over just those members, which sidesteps the approximate index entirely. Up to fifty thousand, we read the ranked list deeper than usual, until the scope is covered. Beyond that, the scope overlaps the normal candidate pool enough that the extra work would not pay.

Did I show you everything?

Which brings me to the question I said nobody ships, and that I now believe is the actual definition of a trustworthy search: can the system tell you whether it has shown you everything?

This matters well beyond user comfort. "Select all 1,240 and apply this rights policy" is only safe if 1,240 is provably the whole set. A search that silently truncated at 1,000 turns a bulk operation into a data-integrity incident that nobody notices for months.

So every freedam search returns a verdict alongside its results: complete, truncated, or unknown. It is computed by extra counting columns in the same query, which measure how many rows each candidate source actually produced against the cap that source was operating under. If a source hit its bound, the result is truncated, and the interface says so, and bulk operations on it warn you. Scoring every member of a small collection is complete by construction. A deep scan is complete only if it finished before its row bound. A new candidate source defaults to unknown until someone proves otherwise. And a result reached through a fallback, a typo correction or a prefix expansion, never inherits the verdict the original query earned.

I want to dwell on a broader rule that this taught us, because it took four separate incidents in a single day to internalise and I think it applies to a lot more than search. A check whose failure mode is silence is not a check. A monitor that reported nothing while the job it watched had died. A tie-break test that stayed green after the tie-break had been stripped out of three layers of SQL. A quality suite whose warnings could not move an exit code. The question that finds these is always the same: if the thing this check watches broke right now, what would it print? If the answer is "the same as when it is fine", you do not have a check. You have a comment with a test-runner icon next to it.

The other ways people ask

Text is one input. Three more matter for images specifically, and I will be brief because each has its own article.

You can search with a picture. "Have we already licensed this?" "Find the rest of this shoot." This uses none of the above: perceptual hashes, looked up through a multi-index scheme, then verified with a vision embedding. Cheap and coarse to generate candidates, expensive and accurate to confirm them. The interesting part is where the textbook algorithm stops working on a real catalogue, and that is a story on its own.

You can search for a person. Faces get their own detector and their own embedding space, in a sidecar that never sends an image off your infrastructure. The design constraint there is not technical: a face has no name until a human gives it one. The machine groups; a person confirms; only then is the name written into the search document, so that photos of Sara at the launch works as an ordinary text query.

And you can search with a sentence. This is where I expected to put a language model, and where I think most people put it in the wrong place. Our assistant does not rank assets. Asking a model to look at 200,000 images per query costs seconds and dollars and gives a different answer twice. Instead it compiles your sentence into the same filter tree the advanced search UI produces, with an explicit intent (replace, refine, add, remove), and that tree runs through exactly the retrieval described above, under the same permissions, with the same completeness verdict. Which means you can see what it decided you meant, correct one filter, and it cannot invent an asset. Language models are excellent at interpreting a query and poor at scanning a corpus. They belong at the front of the pipeline, not in the middle.

How we know any of it works

Every number in this article is a number someone argued about, and the only thing that ends those arguments is a measurement. So we have one: sixty realistic queries with agreed ground truth, run through the production search path against a replica of a real customer library of about 23,000 assets, reporting hard pass/fail plus recall and reciprocal rank. It can only read. It cannot change a setting or rebuild an index.

The first serious run found fifteen hard failures in a search everyone described as pretty good. That was humbling, and it has been consistently useful since, mostly by being right when we were confident. At one point we attributed five recall regressions, with some certainty, to a tokenisation change. Measured, that change had moved four of them by exactly zero: identical recall, identical rank, to fifteen decimal places. The actual cause was document length, the video-versus-photo thing from earlier. Analysis proposes; measurement decides.

Back to the dress

Google found the dress in 2001 by reading the words around the picture. We find a product shot in 2026 by reading the words we manufactured around the picture, plus a few hundred numbers that approximate what a model thinks it looks like. The pixels are still not consulted, and I have stopped finding that disappointing. Thirty years of research closed the semantic gap for the queries that have semantics. The rest of a search engine is plumbing, and the real progress there has not been in looking at images. It has been in being honest about what a search did and did not do.

If I had to compress everything above into advice for someone starting out: spend your effort at index time, because nothing downstream recovers a term the tokenizer never emitted. Run a lexical lane and a semantic lane and fuse their ranks, not their scores. Make your thresholds depend on the shape of the query, and make single words earn a lexical hit. Treat every cap as a correctness problem rather than a performance knob, because if your system has a top-k, some user is silently missing results today. And build the thing that can say "I am not sure I showed you everything", because that is the difference between a search people use and a search people trust.

If you would rather see it than read about it, there is a demo with a seeded library. The interesting thing to do is search for something nobody would have tagged, and then search for something inside a folder and count.