Duplicate detection is the automatic identification of files that already exist in a library — either byte-for-byte identical copies, or near-duplicates: the same image re-saved at a different size, compression level, or format. It typically runs at ingestion, so duplicates are flagged as content arrives rather than discovered years later.

Exact duplicates are easy: a checksum of the file's bytes matches or it doesn't. The interesting problem is near-duplicates, and the standard tool is perceptual hashing — a fingerprint computed from what an image looks like rather than from its bytes. Two files that show the same picture produce similar fingerprints even if one is a resized JPEG export of the other's original TIFF, so the system can say "this looks like something you already have" across format boundaries that checksums cannot cross.

Duplicates matter beyond wasted storage. Every copy is a governance leak: metadata gets corrected on one and not the other, usage rights attach to the original while the stray copy circulates unencumbered, and search returns three versions of one image with no signal about which is canonical. Catching duplicates at the door keeps the library's core promise — one asset, one record, one truth — intact. It is also honest work during migrations, where folder archives routinely turn out to be a third redundant.

In freedam, perceptual-hash duplicate detection runs in the ingestion pipeline, flagging visually identical and near-identical assets automatically.

Related terms: Ingestion, Visual Search, Version Control