Freedam
EngineeringPart 6 of 7 · 18 min read

A face is not a name

Every other kind of search in a digital asset manager fails politely. Ask for a red sneaker and get a maroon boot, and the cost is a second query. This series has spent four articles on that class of problem: the retrieval architecture, near-duplicate detection, twenty languages in one index, and how any of it gets measured.

This one is about the search that does not fail politely. Ask a library who appears in a photograph and a wrong answer is not a wrong result, it is a claim about a person: that they were somewhere, with someone, on some date. The retrieval mechanics are, honestly, the easy part. A cosine distance is a cosine distance. What makes this subsystem different is that the numbers have to be defensible to somebody who is not a search engineer and did not consent to being a data point.

So this article does two things at once. It describes a working pipeline, with the parameters, and it shows the measurement that justifies the one number in it that matters, taken from a real library of 22,944 assets. That number is a similarity threshold, its default is 0.6, and until you have seen the distribution underneath it, it is a magic constant.

Part 1: Three jobs, three costs of being wrong

Before any model, the same question every subsystem in this series has started with: what does correct mean here? For faces there are three answers, and they do not agree.

Three person-search jobs and what each one pays for a mistake THE JOB A FALSE MATCH COSTS A MISS COSTS "Every approved photo of the CEO, for the press." a communications team, in a hurry the wrong person published under someone else's name a second search "Each speaker's own set, from 4,000 frames." an event photographer, delivering a few frames to discard a frame nobody delivers, and nobody notices "Find every image of me. I am withdrawing consent." a former employee, or their lawyer someone else's photo destroyed an image that should be gone, still in the library Only the third job is expensive in both columns, and a system tuned for the first two gets it wrong either way.
The same index, the same distance metric, three incompatible definitions of a good result. The bottom row is the one that decides the architecture.

The first two rows are ordinary retrieval trade-offs, and a threshold dial handles them. The third is not a retrieval problem at all. It is a request that the system be auditable: someone needs to be able to say what the library knows about a person, and a system that answers by guessing is worse than one that answers "only what a human told me".

That is the constraint that shapes everything below. Not accuracy. Auditability.

Part 2: Detection and recognition are different problems

These two words get used interchangeably and they describe completely different acts.

Detection answers "is there a face in this rectangle of pixels, and where". It is geometry. It produces a bounding box and a confidence, it involves no identity whatsoever, and it is as personal as a motion sensor noticing that something moved.

Recognition answers "is this face the same person as that face". It is a claim about a human being, it can be wrong, and when it is wrong it is wrong about somebody.

Systems that treat these as one pipeline stage end up with an architecture where a detection confidence and an identity confidence are the same number, and there is no point in the flow where a human can intervene, because there is no seam to intervene at. freedam keeps the seam. Detection is automatic and produces anonymous geometry. Recognition is a separate step against a set that a person built.

The pipeline, with the seam between anonymous geometry and identity AUTOMATIC · NO IDENTITY IDENTITY W_1024 preview not the original master file InsightFace buffalo_l, 640 x 640 on your own hardware box + score + 512 numbers anonymous, and personal a human names somebody the only step that turns a face into a person a Person, named with a reference set later faces match against that set Everything left of the amber box happens without anyone deciding anything about identity. Everything right of it exists because somebody typed a name. The seam is the product: it is where you can answer "who decided this?" with a person and a timestamp rather than with a model version.
The detector never learns a name. The name never reaches the detector. Everything interesting in this article happens at the join.

The detector is a self-hosted InsightFace sidecar running the buffalo_l model pack at a 640 by 640 detection size, on CPU. It exposes one useful endpoint, /detect, which takes image bytes and returns boxes, detection scores and 512-dimension ArcFace embeddings. There is no third-party API in this path and no image leaves the deployment, which for a subsystem that processes photographs of employees and customers is not a performance detail. It is most of the argument.

The sidecar caps at 50 faces per image; the application caps lower, at 20. Detection runs on its own queue with a five-minute timeout, three attempts and a 10, 60, 300 second backoff, and a failure never blocks the rest of ingestion. A library that will not finish importing because a face detector is unhappy is a worse product than one that imports and finds faces late.

Part 3: What the detector was shown

Article one opened by insisting that everything a search can retrieve was decided upstream, at index time, by code most people never look at. Faces are the cleanest example of that in the entire system, because the upstream decision is a single line and its consequences are unrecoverable.

The detector is not given the original file. It is given the W_1024 preview: the same downscaled rendition the rest of the pipeline uses. The reason is practical, and the code says so bluntly. DAM masters are routinely 40 or 80 megapixels; sending those to a detection sidecar is wasteful, and with a 50 MB upload ceiling it is a timebomb rather than a trade-off.

What it costs is small faces. Here is the measured distribution of detected face sizes on a real library.

Measured detected-face widths and detection scores on a 22,944-asset library DETECTED FACE WIDTH, IN PIXELS OF THE 1024-PIXEL PREVIEW 0 150 300 450 600 750 min 9 10th pct 115 median 247 mean 265 max 708 WHAT WAS FOUND 20,549 images in the library 741 of them contain a detected face (3.6%) 774 faces total: 708 images with one, 33 with two DETECTION SCORE lowest 0.506 · mean 0.760 · highest 0.909 no face was stored without an embedding most images in this library contain no people at all The 9-pixel face is the interesting end. A face that small in the preview was perhaps 35 pixels in the original. A face half that size is simply absent from this table: nothing downstream can find a person the detector was never shown.
Measured on a replica of a real library. The preview decision is invisible in every metric except the one nobody can compute: the faces that are not in the table.

One thing I want to be straight about, because this series' whole value is that it does not assert what it has not measured: I could not measure the cost of the preview decision directly. Doing that properly means running the same images through the detector at full resolution and at W_1024 and comparing, and the replica this data comes from holds the database without the original files. So the paragraph above tells you the shape of what was found and reasons about what that implies, and it does not give you a recall figure for small faces, because I do not have one.

What can be said without measuring is the direction. Downscaling is monotonic: it never reveals a face. Whatever the number is, it is a loss, and it was chosen deliberately in exchange for a detector that finishes.

Part 4: The embedding is a template, not a picture

For each detected face the sidecar returns 512 floating-point numbers, normalised, from the ArcFace recognition head. In the database that is a vector(512) column alongside the bounding box, the detection probability, and a nullable recognition similarity, with an HNSW index over cosine distance restricted to rows that actually have an embedding.

It is worth being plain about what that column is, because "embedding" is a comfortable word that does a lot of hiding. Those 512 numbers are a biometric template. They are not a picture and you cannot look at them, but they identify a person more reliably than most of the metadata in the library, and they are personal data in every jurisdiction that has an opinion on the subject. Treating that column as though it were a thumbnail cache is the single easiest way to build something indefensible.

A cropped 150 by 150 thumbnail is stored too, with 20% padding around the box, JPEG quality 85 and metadata stripped, so a reviewer can see which face a row refers to without opening the original.

The cost is small and worth stating exactly, because "AI feature" and "cheap" are rarely used in the same sentence. On the measured library, the entire faces table including its HNSW index and out-of-line vector storage is 3.3 MB for 774 faces, about 4.3 kB per face. Face data is not what fills a DAM. Pixels are.

Part 5: No clustering, deliberately

Here is the design decision that most distinguishes this subsystem, and it is a decision to do less.

The obvious product move, and the one most consumer photo tools make, is clustering: group the embeddings by similarity, present each cluster as an unnamed person, and invite the user to put a name to it. It demos beautifully. Upload ten thousand photos and the system hands back a wall of strangers, sorted.

freedam does not do it, and the reason is the third row of figure 1.

Clustering strangers versus matching against a human-seeded reference set CLUSTER FIRST, NAME LATER The system proposes identities on its own. Every person in the library becomes a durable entity, named or not, whether or not anyone ever asked a question about them. "Who decided these are the same person?" has no answer but a model version. SEED FIRST, MATCH AGAINST IT Detected faces stay unassigned. A person exists only because somebody created one and attached a face to it. Matching runs only against faces a human already assigned. "Who decided?" answers with a user id and a timestamp. The right-hand model is worse at demos and better at questions. It cannot show a wall of everyone in your library, because it does not know who is in your library. That is the feature: an identity here is asserted by a person, not produced by a clustering parameter.
Both approaches use the same embeddings and the same distance. They differ only in who is allowed to assert that two faces are one person.

Now the nuance, which matters because the summary version of this design is slightly too flattering and a reader who works out the difference for themselves will rightly discount everything else on the page.

The human decision happens once per person, not once per face. Somebody selects a face, gives it a name, and a Person is created. From that moment the person has a reference set, and subsequently detected faces are compared against it automatically: anything above the similarity threshold is assigned without further confirmation. There is also an opt-in bulk match at assignment time, which sweeps existing unassigned faces into the new person in one go.

So the accurate sentence is not "a human confirms every face". It is "a human decides that this person exists and this face is theirs, and the system extends that decision by similarity". That is a meaningfully weaker guarantee than per-face confirmation and a meaningfully stronger one than clustering, and which of those two comparisons you find more important probably depends on whether you are shipping the feature or reviewing it.

It also puts the entire weight of the design on one number.

Part 6: What 0.6 actually means

The threshold is a tenant setting, face_recognition.recognition_threshold, defaulting to 0.6 and clamped to the zero-to-one range. Faces at or above that cosine similarity to a person's reference face are assigned to that person.

A default like that is worth nothing on its own. Published without the distribution it came from, 0.6 is a number somebody liked. So here is the distribution, computed over every possible pair of the 774 faces in the measured library: 299,151 pairs, no sampling.

Cosine similarity across all 299,151 face pairs, showing two separated modes SHARE OF ALL 299,151 PAIRS, BY COSINE SIMILARITY 0 2% 4% 6% 8% four bars clipped at 8%; true values 7.6, 23.3, 29.0, 16.6% -0.2 0.0 0.2 0.4 0.6 0.8 1.0 0.167 highest pair of faces from a single photo 0.6 default valley PAIRS AT OR ABOVE 0.3031,295 0.3531,162 0.4030,814 0.5029,170 0.6023,691 0.7013,016 0.804,391 of 299,151 Two modes with almost nothing between them. The left mode is pairs of different people, peaking just above zero. The right mode is pairs of the same person. The floor between them, near 0.30, holds 133 pairs: 0.04%. A threshold anywhere in that trough behaves the same, which makes the choice safe rather than lucky.
Every pair of 774 real faces, exhaustively, no sampling. The four tallest bars are clipped to keep the right-hand mode legible; their true percentages are printed beside them.

The shape is the whole argument. ArcFace does not produce a smear of similarities that you have to cut arbitrarily; it produces two populations. Different people cluster just above zero, with a median pair similarity of 0.029. Same-person pairs cluster around 0.65 to 0.70. Between them, at about 0.30, is a trough holding 0.04% of all pairs.

But calling the left mode "different people" and the right mode "same person" is an interpretation, and interpretations are how articles like this go wrong. So here is a check that needs no labels at all.

Two faces detected in the same photograph are, essentially always, two different people. That gives a set of pairs which are known to be non-matches by construction, without anyone identifying anybody. There are 33 such pairs in this library. Their similarities run from -0.021 to 0.167, with a mean of 0.057. Not one of them reaches 0.35. Not one of them comes within a factor of three of the 0.6 threshold.

That is a small sample and I will not pretend otherwise. It is 33 pairs, from one library, in one visual style. But it is a sample that cannot be wrong about its own labels, and it lands exactly where the left mode is, which is the useful kind of corroboration.

One more thing falls out of the same table, unasked. 1,647 pairs have a cosine similarity of exactly 1.0, and every one of them spans two different assets. Identical templates from different rows means the same photograph is in the library more than once. A face embedding turns out to be an accidental duplicate detector, which is a pleasing echo of the second article, and a reminder that a library's real content is always messier than its schema suggests.

Part 7: The threshold does less than you would think

Given a distribution that clean, the obvious question is what you actually buy by moving the dial. The answer surprised me, and it reframes what the setting is for.

Lowering the threshold barely changes reachability but sharply increases links drawn FACES WITH AT LEAST ONE MATCH, OF 774 PAIRS LINKED 0.80 728 4,391 0.70 747 13,016 0.60 758 23,691 0.50 763 29,170 0.35 766 31,162 FROM 0.60 DOWN TO 0.35 8 more faces become reachable at all. That is 1.0% of the corpus. THE SAME CHANGE draws 7,471 additional links between faces. That is a 32% increase in assertions made.
Reachability saturates almost immediately. Everything below the first few hundredths of the dial is spent on the density of the graph, not on whether anyone can be found.

At 0.6, 758 of 774 faces have at least one match somewhere in the library. Drop the threshold all the way to the valley floor at 0.35 and that becomes 766. Eight faces. Meanwhile the number of pairs the system is willing to call the same person rises from 23,691 to 31,162.

The threshold is not really a recall control. It is a control on how much the system is willing to assert. Loosening it does not meaningfully help people be found, because by the time you are anywhere near the trough, almost everyone who appears more than once has already been linked. What loosening does is manufacture claims, and every claim is a chance to be wrong about a person.

Which reframes the default. 0.6 sits well above the trough, roughly twice the valley floor, on the rising edge of the same-person mode. Read as a recall setting, that looks needlessly timid, and you could "improve" it by moving it down to 0.35 where the maths says the populations separate. Read as an assertion budget, it is the right shape: buy the reachability, which is nearly free, and decline the extra 7,471 assertions, which are not.

The sixteen faces with no match at any threshold are worth a sentence too. Those are people who appear exactly once. No similarity setting will ever connect them to anything, and a system whose interface implies that everyone is discoverable is lying to a user about a fifth of a percent of its own data.

Part 8: A reference set grows, and so does the past

One more mechanism, because it has a consequence that is easy to ship without noticing.

Assigning a face to a person makes that person's reference set larger, and a larger reference set can match faces that were previously unmatchable. A face photographed in profile might miss the one frontal reference it was compared against, then match comfortably once a profile shot has been added. So after any assignment, the system re-evaluates unassigned faces in the neighbourhood of the newly assigned one, and a separate command sweeps every unassigned face in the library against every known person.

The result is that a person in this system is not a fixed query. It is a set that grows, and it pulls the past in behind it. Images that returned nothing for a person last month can return something this month, with no new photographs uploaded, no re-detection, and nobody confirming anything. Only the reference set changed.

That is genuinely good behaviour for the event photographer and the communications team. It is behaviour that a data-protection reviewer will want to know about explicitly, because "the system's answer about this individual changed and no human touched it" is exactly the sort of thing that needs to be true on purpose rather than by accident. It is on purpose. It should be documented in whatever record you keep, and it is a reason to prefer a conservative threshold: retroactive assignment at a loose threshold reaches backwards through the entire library at once.

Part 9: Person search is not a separate engine

The last part is the one that ties this article back to the four before it, and it is deliberately anticlimactic.

When a person has a name, that name is written into the asset's search document: the same per-language text document from the first article, built by the same builder, indexed by the same per-language BM25 index, fused with the same vector scores by the same reciprocal rank fusion. Names are de-duplicated case-insensitively, and the field carries a configurable weight like every other field, so an organisation that considers people more important than filenames can say so numerically.

Faces without a named person contribute nothing at all. An unnamed face is not in the search document, is not in the dictionary, and cannot be found by any query, because there is no text to find. Anonymity here is not a policy applied at query time that someone might forget to apply. It is the absence of an index entry.

The consequence is that searching for a colleague's name uses the machinery this whole series has been describing, with every property it has: the recall guarantees under caps, the completeness reporting, the multilingual merge, the panel that measures it. There is no separate face-search subsystem to reason about, no second relevance model, no parallel ranking to keep consistent with the first one.

That is the argument for putting the seam where it is. Detection produces geometry. A human produces a name. And a name is just text, which the rest of the system already knows how to find.

What we would tell anyone building this

  • Separate detection from recognition in the architecture, not just the vocabulary. If there is no seam between anonymous geometry and asserted identity, there is nowhere for a human decision to live, and no way to answer "who decided this?".
  • Design for the withdrawal request, not the demo. The job with an expensive answer in both the false-match and the miss column is the one that fixes your architecture. Everything else is a threshold.
  • Publish the distribution or do not publish the threshold. A similarity default with no impostor and genuine distributions behind it is a magic constant, and it will be copied into someone else's system where it means something different.
  • Find labels your data gives you for free. Two faces in one photograph are two different people. That single structural fact produced a validation set with no annotation, no cost and no possibility of a labelling error.
  • Check whether your threshold is a recall control at all. Ours moves reachability by 1% and assertion volume by 32% over the same interval. Those are different products and the setting looks like the first one.
  • Know that a reference set makes identity retroactive. When adding a face changes what the system says about images uploaded years ago, that is a real behaviour with a real audience, and it should be a documented choice rather than a discovered one.
  • Decide early what a template's lifecycle is attached to. Giving biometric data its own independent lifecycle and tying it to something you already control are both defensible. Drifting into one of them without choosing is not.
  • Say what you cannot do. We could not measure the cost of detecting on a downscaled preview, so this article does not contain that number. An article that invented one would be worth less than an article that admits the gap.

Where this lives in the product

Face detection and person search run inside the same deployment as everything else: the InsightFace sidecar, the PostgreSQL instance holding the vectors, the queue workers. No image and no template is sent to a third-party service, which is a large part of what makes the whole system practical to self-host and is the difference between a feature a legal team can approve and one they cannot.

Once a person is named, they are searchable everywhere search reaches: the gallery, the REST API, the TypeScript SDK, and any AI agent connected over MCP. Who is allowed to run those searches is an access control question rather than a face question, and it is answered by the same permission model that governs every other asset in the library.

Every number in this article comes from a single library, with all 299,151 pairs compared exhaustively rather than sampled. One library is one library: a different mix of photography, lighting and framing would produce a different distribution and quite possibly a different sensible threshold. The method is the transferable part, not the constant. Compute your own pairs, find your own trough, and use the pairs your own data labels for free to check which mode is which.

Next in this series: conversational search, where a language model is put in front of all of this and given exactly one job that has nothing to do with looking at pictures.

Keep reading

What it actually takes to build a good image search
The architecture of image search
Finding the same image twice
Searching a library in twenty languages