The chat that never sees your library

There is a version of AI search that every team builds first, and it is the wrong one. The user types a sentence, the sentence goes to a language model, the model is handed some assets, and the model decides which ones are good. It demos beautifully on a library of forty. It falls apart at four thousand, and the way it falls apart is instructive: not with an error, but with a plausible answer that happens to be a subset of whatever was in the context window.

freedam's conversational search is built on the opposite premise. The model never sees a single asset. It has no access to the library, no ability to rank, no opinion about relevance. Its entire job is to turn a sentence into a filter tree, and then it is done. Everything after that is the same retrieval engine described in the architecture article: BM25 and pgvector fused inside one PostgreSQL query, under the same access rules, returning the same totals as the gallery.

This article is about what that boundary buys you, what it costs, and the parts nobody writes down: how conversational state survives when the transcript does not, why the assistant's reply is a prediction rather than a report, and the three places where our merge semantics quietly do the wrong thing.

Part 1: Interpretation is cheap, scanning is not

The two jobs hidden inside "search with AI" have opposite cost curves, and conflating them is what breaks the naive design.

Interpreting a query is a fixed-cost, high-skill job. "Something calm for the newsletter header, landscape, not the ones we used last spring" contains a mood, an aspect ratio and a temporal exclusion, and unpacking it is exactly what a language model is extraordinary at. It costs one model call regardless of whether the library holds four hundred assets or four hundred thousand.

Scanning a corpus is a fixed-skill, high-volume job. It requires no judgement and enormous throughput, and its cost is linear in the library. It is what a database with the right indexes has been optimised for across five decades.

The consequence that matters most is not cost, it is honesty. A retrieval engine can tell you it found 1,247 matches and that the number is exact. It can tell you it hit a candidate cap and that the number is a lower bound, which is the completeness reporting the rest of the search stack is built around. A model handed a page of assets can tell you neither, because it never knew what it was not shown. A system that cannot distinguish "there are none" from "I did not look" cannot be the basis of a bulk operation.

So the boundary is drawn in exactly one place. The model owns language. The database owns the library. Nothing crosses.

Part 2: What the model is actually asked for

One turn produces exactly one JSON object, and it has three keys.

{
  "intent": "refine",
  "rule_group": {
    "type": "group",
    "operator": "and",
    "rules": [
      {"field": "full_text_search", "operator": "hybrid_search", "value": "sunset"},
      {"field": "asset.aspect_ratio_orientation", "operator": "equals", "value": "landscape"}
    ]
  },
  "explanation": "Narrowing to landscape shots with sunset colours."
}

rule_group is not a bespoke chat format. It is the same rule tree the gallery's advanced filter editor builds by hand, the same one saved searches serialise, and the same one the rule compiler turns into SQL. That choice is the whole design in one line: the model's output is a data structure the product already knew how to execute. There is no chat-specific query language, no chat-specific ranking, no second retrieval path that can drift away from the first one.

intent says what to do with the tree relative to the conversation so far. explanation is the only natural language the user ever sees, and the system prompt pins it to one short sentence in the user's own interface locale, which matters more than it sounds and is where the multilingual work shows up again: field names, operator names and intent values stay in English because they are identifiers, while the sentence around them is generated in French or Simplified Chinese.

Three settings turn a generative model into something closer to a compiler: temperature 0.1, a json_object response format, and, when the configured model supports it, OpenRouter's Exacto routing variant, which pins the request to providers vetted for structured output. The default model is a mid-size open-weight one, openai/gpt-oss-120b, because the task is translation into a schema rather than reasoning, and the schema is enforced downstream anyway.

Part 3: The prompt is mostly a database dump

Here is the part that surprises people who assume prompt engineering means writing prose. On a development tenant holding 22,944 assets and 48 metadata definitions, the system prompt for a search turn measures 36,790 characters across 646 lines, and 16,646 of those characters, roughly 45%, were generated from that tenant's own database seconds earlier.

The fixed part, 19,840 characters, is the instruction skeleton: the output schema, the six intents, the search-method priority order, the worked examples, and a list of things not to do that grew one entry at a time from watching real failures. The generated part is a catalogue of the tenant's business vocabulary, assembled per asset class, and it is what makes the same prompt behave completely differently for a furniture retailer and an aviation parts supplier.

Three decisions inside that generated section are worth stealing.

Sample the values, not just the schema. For every text field, the prompt carries the twenty-five most frequent values actually present in the library. For every number and date field, the observed minimum and maximum. For every vocabulary field, up to fifty real term labels. A model that has been told a field named product_designer exists will guess at it; a model that has been shown that its values look like Henning Koppel and Louise Adelborg will map a surname onto it correctly on the first try.

Never make the model guess an identifier. Vocabulary terms are foreign keys, and a model asked to produce a term ID will invent one that parses. So the operator set includes vocab_text_equals and vocab_text_in, which take human text, and the resolution to an ID happens afterwards in the compiler using PostgreSQL trigram similarity with a 0.45 threshold and an exact case-insensitive match tried first. The model contributes the thing it is good at, which is deciding that "Alfredo" in this sentence is a designer. The database contributes the thing it is good at, which is knowing that the designer is Alfredo Häberli. This is the same division of labour as the whole architecture, applied one level down.

Let the retrieval over your own schema be honest about being weak. Six fields carry a high semantic priority flag and are always present. Everything else is chosen by a scoring function that lowercases the query, drops words of three characters or fewer, and counts substring hits against each field's label, key and keyword list. Measured against two realistic queries on that tenant, it contributed two and three fields respectively, out of 86. The cap of 25 has never once been the binding constraint. It works because the always-on six plus the full metadata catalogue already cover the ground, but it is worth naming clearly: this is a keyword matcher standing where a retrieval system would go, and its adequacy is a property of our field count, not of the design.

Part 4: Validation, and the retry that is a conversation

A model that returns valid JSON has cleared the lowest bar in the system. The interesting failure is the one that parses: a well-formed tree referencing asset.file_type, a field that has never existed.

So validation runs in two layers. The first checks structure: the three required keys, an intent drawn from the permitted six, a rule_group carrying type, operator and rules. The second walks the tree recursively and checks every leaf's field name against the same registry the gallery's filter editor reads. An unregistered field is a hard rejection, not a warning, because a filter on a nonexistent field either fails to compile or, worse, compiles into something that silently matches nothing.

When rejection happens, the errors do not become an exception. They become the next user turn:

ERROR: Your previous response had validation errors:
Invalid field 'asset.file_type': The field 'file_type' does not exist.
Use 'asset.mime_type' (for MIME types like 'image/jpeg') or
'asset.extension' (for file extensions like 'jpg') instead.

Please correct these errors and provide a valid response
following the exact schema.

The invalid response is appended as the assistant's turn, the error message as the user's, and the whole thing is sent back. The model is now debugging its own output with a compiler error in hand, which is a task it is markedly better at than getting it right blind. The most common mistakes carry hand-written remediation text, because "field is not registered" tells the model it was wrong and the message above tells it what to do instead.

Two retry loops are nested. Transport failures get one retry after a two second backoff. Validation failures get one retry with feedback. The worst case before the user sees a fallback message is therefore four model calls, and with a 45 second per-call timeout that is an upper bound that comfortably exceeds most reverse proxy patience. We have not measured how often the outer loop is reached in production, and the honest reading of that is that the bound is a design fact rather than an observed one.

Below the boundary, one more normalisation runs unconditionally, and it exists because of a real class of near-misses. Colour values arrive from models in every shape a human might write: #f00, FF0000, {"hex": "#ff0000"}, sometimes with a tolerance and usually without. The leaf constructor expands three-character hex to six, uppercases it, restores the leading hash, and supplies a default tolerance of 24 when none was given. A parser that is strict about structure can afford to be generous about spelling, and the alternative, another retry round trip because the model wrote #f00, is a second of latency spent on nothing.

Part 5: The conversation is a tree, not a transcript

This is the part that took longest to get right, and it is where most chat-search implementations quietly break.

The obvious design is to make the transcript the state: keep the last several turns, send them up, let the model reconstruct the user's intent every time. It fails in two ways at once. It drifts, because reconstructing an accumulated filter set from prose is lossy and the loss compounds. And it makes the conversation unexportable, because there is no moment at which "what the user is currently searching for" exists as a thing you can hand to the gallery, save, or share.

In freedam the state is the rule tree, stored on the session, updated once per turn. The transcript is context, capped at the last ten messages, and if it were dropped entirely the current search would survive intact. The model is told the current tree in full on every turn, and is asked not to restate it, only to describe the delta and label the delta with an intent.

The payoff is concrete. Because the state is a tree, the tree can leave. The chat page's "open in gallery" hands the accumulated filters to the gallery's rule editor, where the user gets checkboxes and dropdowns for everything the conversation built, and can then save it, share it, or attach it to an automation. A conversation that terminates in a data structure is a conversation the rest of the product can inherit. The handoff itself is a shape adaptation rather than a translation, because top-level bare filters have to be wrapped into a group for the editor, and it is worth noting that filter trees in this codebase are never flat: several layers of wrapping are added by different callers, and any code that inspects a tree by looking only at the root's direct children will match nothing in production while passing every test whose fixture was built flat by hand.

Now the honest part. Three sharp edges live in this merge, all of them found by reading the code rather than by a user complaining, which usually means the users have been working around them.

A refine that changes a value is silently dropped. Refine deduplicates on field plus operator, ignoring the value. So a session already holding width > 1000 that is refined with width > 2000 keeps the first one. The user sees a confident sentence about narrowing, and nothing narrows. The dedup rule is right for its intended case, a model helpfully repeating an existing filter, and wrong for the adjacent one.

A remove requires the model to echo the value exactly. Removal matches on field, operator and value, so "drop the width filter" only works if the model reproduces 1000 verbatim from the state it was shown. It usually does, and when it does not, the failure is silent.

A filter nested inside a group cannot be removed at all. Removal walks only the top level of the tree by design. Every palette search the prompt encourages produces exactly such a nested OR group of three or more colour rules, so "forget the sunset colours" cannot succeed through this path. Reset works; targeted removal does not.

None of the three throws. All three end in a cheerful explanation describing a change that did not happen, which is the worst available failure mode and precisely the one the checks-that-cannot-fail discipline exists to catch. They are written down here because writing them down is the first step to fixing them.

Part 6: One word, two meanings, three states

"Landscape" is an orientation. "Landscape" is also a subject. In a photo library both readings are common, and a system that silently picks one is wrong roughly half the time without ever saying so.

The handling is small and worth copying. The word is tracked in the session as a three-state memory, stored alongside the rule tree.

Two properties make it work. It is inferred from behaviour: the resolution is derived by inspecting the merged tree for an orientation filter or for the word appearing inside a search value, never from the model claiming which reading it chose. And it is sticky: once resolved, the interpretation is asserted in every subsequent prompt, so turn seven does not quietly flip the meaning that turn two established.

When the term is still unresolved, the model is instructed to return the clarify intent, which is the one path through the handler that changes no filters and runs no search. The unknown marker is persisted anyway, so the system remembers that it asked. The cost of a clarify turn is one model call and zero database queries, which is the correct price for a question.

Two words are handled this way today, landscape and portrait. The mechanism is general and the vocabulary is not, and the reason is honest rather than principled: those are the two we observed. A per-tenant ambiguity list learned from clarification outcomes is the obvious next version and does not exist.

Part 7: The reply is written before the search runs

This ordering is not a bug, but it is the source of the most subtle correctness problem in the whole feature, and it took a while to see clearly.

The model produces its explanation at step 2. The search executes at step 6. Whatever the sentence says, it was written before the search existed, by a component that will never learn what happened. If the strict query returned nothing and the retrieval layer widened it with typo tolerance or a prefix match, the assistant is still cheerfully describing the search it asked for.

The fix is not to call the model a second time to narrate the result, which would double the cost of every turn to improve one sentence. The search result already carries a structured account of how it relates to what was asked: a mode of strict, typo or prefix, the original term, and the term the fallback actually ran. The interface renders that as its own notice under the reply. The chat is currently the only surface in the product that reads it; the gallery and the REST API expose only the separate "did you mean" hint, which is a suggestion generated alongside the search rather than a description of the query that ran. Those two are easy to confuse and they are not the same value.

The generalisable rule: when a language model speaks before the deterministic system acts, the interface owes the user a channel the model cannot write to.

The same reasoning governs the preview itself. It would have been easy to give chat its own lightweight search. It does not have one. The preview goes through the identical entry point the gallery's advanced filters use, with the same profile, the same relevance sorting rule, the same candidate caps, the same completeness reporting, and the same access rules resolved from the session's own user. It fails closed: a session whose user cannot be resolved raises rather than falling back to an unfiltered search, because the alternative is a preview that shows someone assets they are not allowed to see. A pair of parity tests pins chat and gallery to the same assets, the same totals and the same fallback metadata for identical rules, running against real PostgreSQL rather than mocks, including under an access rule that hides part of the library.

Each of these paths also carries its own caller identity into telemetry, chat_preview, chat_diagnostic, chat_suggestions, so a search executed on behalf of a conversation is legible in the logs as such rather than blending into gallery traffic.

Part 8: Explaining a zero is worth five queries

The hardest moment in conversational search is not the good answer, it is the empty one. "No results" from a keyword box is a familiar disappointment. "No results" from something that just spoke to you in a sentence reads as a failure of the conversation.

So a zero-result turn spends real money on explaining itself. Each filter in the tree is re-run in isolation, counting only, through the same access-filtered pipeline, up to five filters. The user gets a breakdown: designer Henning Koppel, 412 matches. Ingested after 1 October, 88 matches. Portrait orientation, 0 matches. The restrictive one is now visible, and the conversation has somewhere to go.

The suggestions built on top read those counts. If several filters match individually but their conjunction does not, the system proposes the disjunction. If a date range is present, it proposes widening it. If a colour or dimension constraint is present, it proposes relaxing it. And when a search returns a small but non-zero number, between one and four, a different generator samples the AI tags of the results and offers the frequent tags that are not already in the query, which is expansion by example rather than by rule.

Turn outcome	Model calls	Database searches
Clarify: the model asks a question	1	0
Normal: five or more results	1	1
Few: one to four results	1	2
Zero, with up to five filters	1	up to 6
Malformed JSON, recovered on retry	2	1
Transport failure, recovered on retry	2	1
Exhausted every retry	4	0

Two things about that table. The zero-result row is the most expensive in database terms and the cheapest in model terms, which is the right way round: counting is what the database is for, and a bad answer explained is worth more than a bad answer apologised for. And the last row is the one to design against, because four model calls that end in a generic fallback message is the maximum spend for the minimum value.

A separate cost lives before all of this. Every message passes a topic check, a second, much smaller model call that decides whether the question is about assets at all, keyed by a hash of the message text and cached for an hour so repeated phrasings are free. It is a blunt instrument and it is deliberately fail-open: if the check errors or no key is configured, the message goes through. A guard that blocks real work when it breaks is worse than the abuse it prevents.

Around that sit the ordinary controls: twenty messages per minute per user, a 2,000 character message cap, sessions that expire after thirty days, per-message token and cost recording that rolls up to a session total, and a kill switch that any automated cost monitor can pull, which disables the feature for twenty-four hours through a cache flag without a deploy.

Part 9: What we would tell anyone building this

The transferable version, stripped of our specifics.

• Put the model in front of your search engine, never inside it. Interpretation costs one call regardless of corpus size; ranking costs the corpus. Only one of those should be paid per query, and it is not the one most first drafts pay.
• Make the model emit a structure your product already executes. If the conversational path has its own query language, it will have its own bugs, its own ranking, and eventually its own idea of what your data means. Ours emits the same rule tree the manual filter editor builds.
• Validate against your live registry and hand the errors back as a turn. Structural validation catches nothing interesting. Checking every field name against the same registry the rest of the app reads is what stops confident nonsense, and a model given a specific correction fixes itself far more reliably than one told it was wrong.
• Keep conversational state in the structure, not in the transcript. State that lives in prose drifts and cannot be exported. State that lives in a tree can be handed to a filter editor, saved, shared, or attached to an automation, and it survives the transcript being truncated.
• Generate most of your prompt from the tenant's own database, and sample values rather than listing schema. Ours is 45% generated. A model shown that a field's real values look like Henning Koppel maps a surname onto it; a model told only that the field exists guesses.
• Never ask a model for an identifier. Give it text operators and resolve to keys in your compiler with fuzzy matching. Language is the model's job; identity is the database's.
• Assume the model's summary is a prediction, and give the interface its own channel. Anything discovered after the model speaks, a widened query, a truncated total, a fallback, has to reach the user through a component the model cannot write.
• Run the same search everything else runs, under the same permissions, and tag the caller. A conversational surface with its own retrieval path is a second search engine that will silently disagree with the first.
• Spend queries on explaining an empty result. Per-filter counts turn a dead end into the next turn, and they are cheap in exactly the way a second model call is not.
• Write down what your merge semantics cannot express. Ours cannot update a filter's value through refine, cannot remove a nested one, and reports success in both cases. Those are the failures a user never files a ticket about, because the system sounded certain.

Where this lives in the product

Conversational search is one surface over the retrieval engine described across this series, not a parallel system. The tree it builds executes as the same fused BM25 and pgvector query as everything else, respects the same access control rules, resolves the same controlled vocabularies, and is measured by the same relevance instrument. Nothing about the model call is privileged. Turn the feature off and every search it can express is still expressible by hand in the gallery's filters.

The same architecture is what makes the MCP integration coherent rather than duplicative. An external AI agent connecting over MCP is doing exactly what the built-in chat does, translating language into a query and letting PostgreSQL retrieve, and it goes through the same entry point with its own caller identity. The chat is not the AI feature; it is one client of a search engine that was designed to be driven by a program.

The limits are worth stating as plainly as the design. The numbers in Part 3 come from one development tenant with 22,944 assets and 48 metadata definitions, and prompt size scales with a tenant's metadata catalogue, so a library with several hundred definitions will produce a substantially larger one, which is a growth curve we have not yet had to bound. The three merge defects in Part 5 were found by reading and are not yet fixed. The field selector is a substring matcher. And we have no production measurement of how often the validation retry fires, which means the most interesting number in the whole feature, how often a language model gets a schema this constrained wrong on the first attempt, is one this article cannot give you. That gap is the next thing to instrument, and an article that invented the figure would be worth less than one that admits it.