Exact duplicates and near-duplicates
| Kind | Example | How it is detected |
|---|---|---|
| Exact duplicate | The same file uploaded twice, or copied from one shared drive folder to another | A checksum of the file's bytes (for example SHA-256): identical files have identical checksums |
| Near-duplicate | A JPEG export of a TIFF original, a resized web version, a recompressed copy from an email | A perceptual hash: a fingerprint of what the image looks like, which stays similar when the file changes |
| Crop or variant | A square crop for social media, a slightly different colour grade | Perceptual hashes of parts of the image, or vision embeddings |
| Similar image | Another frame from the same shoot | Not a duplicate; visual search finds these |
A checksum changes completely when a single byte changes, so it cannot tell that a resized JPEG and its TIFF original show the same picture. A perceptual hash reduces the image to a small grid of brightness values and encodes its structure, so two files that show the same picture produce fingerprints that differ in only a few bits. The number of differing bits (the Hamming distance) measures how close they are.
Why duplicates cause problems
- Metadata diverges. Someone corrects the caption or keywords on one copy, and the other copy keeps the old version.
- Rights get lost. The usage rights and expiry date are recorded on the original, while a copy without them goes into a partner kit after the licence has ended.
- Search results are cluttered. Three versions of the same product photo appear, with nothing to show which one is approved.
- Reports are split. Downloads and usage are spread across copies, so no copy shows the real figures.
- Migrations carry the clutter over. Libraries moved from shared drives bring every copy with them unless duplicates are removed during the import.
Resolving a duplicate
When a new upload matches an existing asset, there are three sensible outcomes:
- Discard the upload and keep the existing asset with its metadata and history.
- Replace the existing file with the upload, for example when the upload is a higher-resolution version. The existing record keeps its metadata, collections and links. See version control.
- Keep both, when the match is wrong or the two files are deliberately separate, such as a crop made for one channel.
For a clean-up of the whole library, keep the best copy in each group (usually the highest resolution) and move the others to the trash rather than deleting them permanently, so a wrong decision can be reversed. The DAM audit checklist covers duplicates as part of a wider review.
Duplicate detection in Freedam
- Every upload is checked for exact copies by its SHA-256 checksum, and every image is checked for near-duplicates by perceptual hashing.
- A duplicate is flagged for review, not blocked. The uploader compares the two files and chooses Delete Uploaded, Keep Both or Replace Existing, which replaces the file and keeps the existing asset's metadata.
- All the exact duplicates in an upload batch can be deleted in one step.
- Near-duplicate matching compares the whole image, its centre and its four quarters, together with a vision embedding, so crops of an existing image are recognised.
- The asset page of each image lists its near-duplicates in the asset library.
- Administrators can scan the whole library for exact and near-duplicates. Resolving a group keeps the copy with the highest resolution and file size, moves the others to the trash and adds the kept asset to their collections.
Details: Duplicate scan in the Freedam docs.
Frequently asked questions
What is a perceptual hash?
A short fingerprint computed from what an image looks like rather than from its bytes. Two files that show the same picture get similar fingerprints even when their format, size or compression differ. pHash, based on a discrete cosine transform, is the most widely used method.
Does duplicate detection work for videos and documents?
Exact detection by checksum works for every file type. Near-duplicate detection is mainly used for images, because a perceptual hash describes a picture. A video or document re-exported with different settings has a different checksum and is not caught as an exact copy.
Should duplicates be deleted?
Exact duplicates, yes, once you have checked which copy holds the better metadata. Near-duplicates need a look first: a crop or a colour variant may have been made on purpose for a channel or a partner.
Related terms
See duplicates caught on arrival
Nordi's library checks every upload against the files it already holds. Open an image in the demo to see its near-identical matches.
You enter Nordi’s workspace as an administrator. No card, no setup.