Duplicate Scan

The Duplicate Scan feature allows administrators to maintain a clean and efficient library by identifying redundant assets. It uses advanced detection to find both byte-for-byte identical files and visually similar images that may have different filenames or metadata.

By scanning your library, you can reduce storage costs, eliminate user confusion caused by multiple versions of the same file, and ensure that only the highest-quality "best" version of an asset is available for use.

Page Overview

• Purpose: To detect and resolve duplicate assets across the entire library or specific subsets of data.
• When to use it: Perform regular maintenance to clean up bulk uploads, or run targeted scans after migrating new collections into the system.
• What you can do here:
  • Define specific filters to limit the scan to certain asset classes or properties.
  • Monitor real-time progress of the scanning engine.
  • Compare duplicate assets side-by-side.
  • Bulk-delete duplicates while preserving the highest-quality version.
  • Dismiss false positives to keep them in the library.

Page Layout

• Top bar: Displays the page title, a brief description, and a status badge showing the number of groups awaiting review once a scan is complete.
• Main area: This dynamic region changes based on the current state:
  • Filter Section: Shown when starting a new scan; contains the rule editor and asset counter.
  • Progress Section: Shown during an active scan; displays a progress bar and live statistics.
  • Results List: Shown after a scan completes; displays cards for every duplicate group found.
• Comparison Dialog: A pop-up window used to view detailed metadata and large previews of two assets side-by-side.

Main Features

Asset Filtering

• What it's for: Narrowing down the scan to specific assets rather than the whole library.
• Typical use: Scanning only "Photography" or assets within a specific "Marketing" collection.
• Result: The system calculates the number of matching assets before you begin, ensuring you only scan what is necessary.

Visual Similarity Detection

• What it's for: Finding assets that look the same but aren't identical files (e.g., a resized version or a different file format).
• Typical use: Identifying where a high-res TIFF and a low-res JPEG of the same photo are both taking up space.
• Result: Assets are grouped with a "Similar" badge and a confidence percentage.

One-Click Resolution

• What it's for: Quickly cleaning up groups of duplicates.
• Typical use: Deleting all inferior versions of a file while keeping the "Best" version.
• Result: The redundant assets are removed from the system, and the group is cleared from your review list.

Detailed Feature Documentation

Filter Assets

• Purpose: To define the scope of the duplicate search.
• Where to find it: In the main area under the "Filter Assets" card.
• What you'll see: A rule builder where you can add conditions based on asset properties, business data, or ownership.

How to use it:

1. Click the rule editor to add a new filter (e.g., "Asset Class is Image").
2. Observe the Counting assets... indicator.
3. Verify the total number of assets that match your filters.
4. Click Search for Duplicates to begin.

Duplicate Group Review

• Purpose: To evaluate found duplicates and decide which to keep.
• Where to find it: In the "Duplicate Groups" list after a scan completes.
• What you'll see: Individual cards showing a "Best" asset (highlighted in green) and its detected duplicates.

How to use it:

1. Review the thumbnails; the "Best" asset is chosen based on dimensions and file size.
2. Hover over a thumbnail to see the filename and dimensions.
3. Click Compare to see a detailed side-by-side view.
4. Click Delete Duplicates to remove all but the best version, or Dismiss to keep everything.

Complete Workflows

Workflow: Targeted Library Cleanup

• Goal: Find and remove duplicates within a specific category of assets.
• Prerequisites: Administrative permissions to delete assets.

Steps:

1. Navigate to Duplicate Scan in the Administration menu.
2. In the Filter Assets section, create a rule (e.g., "Collection equals 2023 Brand Guidelines").
3. Wait for the asset count to update, then click Search for Duplicates.
4. Once the scan reaches 100%, click Compare on the first result card to verify the match.
5. Click Delete Duplicates to remove the redundant files.
6. Repeat for remaining groups or click New Scan to start over.

• Expected result: Redundant assets are deleted, and the "groups to review" badge count decreases.
• If it doesn't work: Check if the assets are locked or if you have sufficient permissions to perform deletions.

Workflow: Cancelling a Long-Running Scan

• Goal: Stop a scan that is taking too long or was started with incorrect filters.
• Prerequisites: A scan must be in "Processing" state.

Steps:

1. Locate the Scanning for Duplicates card while a scan is active.
2. Click the Cancel button (stop icon) next to the percentage display.
3. Confirm the status changes to Scan Cancelled.
4. Click Start a new scan to return to the filter screen.

• Expected result: The scanning process stops immediately, and no results are generated.

How it works

Freedam runs a duplicate scan in two clearly separated passes so you see results progressively:

1. Exact-match pass (0–50% of progress) — Freedam compares the unique file fingerprint that every asset carries once it has finished uploading. Files with an identical fingerprint are byte-for-byte duplicates, and they are grouped with a confidence of 100%.
2. Visual-similarity pass (50–100% of progress) — For the remaining assets, Freedam compares AI image embeddings against the rest of the library in small batches to find look-alikes. Only pairs that recognise each other in both directions are kept, which prevents unrelated photos from being lumped into the same group just because they share a theme.

When it picks the "Best" version of a group, Freedam sorts by pixel resolution first and file size second, so the highest-quality copy is always kept by default.

Who can use it

• The Duplicate Scan page is only visible to administrators who have been granted the "Access and run library-wide duplicate asset scans" permission.
• Deleting the duplicates in a group additionally requires the usual asset-delete permission.
• Deleted duplicates are soft-deleted: they land in the trash and can be restored within 30 days before they are purged permanently.

Limits and guardrails

• One scan at a time per user: If a scan is already pending or running, starting another one is blocked with "A scan is already in progress." Cancel or wait for the current one to finish first.
• Similarity confidence thresholds: Freedam only keeps visual matches above a confidence of 70% for the normal case. Matches that rely purely on the AI image model (without the fingerprint sanity check) need at least 65%, and very conservative crop-aware matches need at least 30% plus an additional structural check on the asset's thumbnail. This is why you almost never see a spurious "Similar" group for unrelated photos.
• Processing window: A single scan has up to 30 minutes of server time. If you are scanning hundreds of thousands of assets, narrow the filters or run multiple focused scans rather than one giant sweep.
• Session lifespan: Scan results are kept for 1 hour after they finish. If you come back the next day, start a new scan — the previous results will have expired.
• Auto-recovery: If the scan crashes for any reason, it is marked as failed, the error is recorded server-side, and the scan can simply be restarted — no partial data is left behind.

What happens behind the scenes

• Live progress: While the scan runs, Freedam updates the progress bar several times a minute so you can see both how many assets have been inspected and how many duplicate groups have been found so far.
• Cancellation is immediate: Clicking Cancel marks the scan as cancelled; the background worker checks for the cancel signal between chunks and stops at the next checkpoint, so no more deletions or comparisons happen.
• Delete vs. dismiss: Choosing "Delete Duplicates" soft-deletes every asset in the group except the "Best" one and records who did it and when. Choosing "Dismiss" simply marks the group as reviewed so it disappears from the review list without touching any files.

Tips and Best Practices

• Trust the "Best" Badge: The system automatically flags the asset with the highest resolution and largest file size as the "Best" version.
• Start Small: If you have a massive library, use filters to scan one collection or asset class at a time to make the review process more manageable.
• Check Confidence: For "Similar" matches, check the confidence percentage. High percentages (95%+) are almost always duplicates, while lower percentages may just be photos from the same photoshoot.
• Mistakes are reversible: Because deleted duplicates go to the trash first, you have 30 days to restore anything that turned out not to be a duplicate.

Troubleshooting

Issue: Scan Failed or Cancelled

• Symptoms: An error message appears with a red border, or the scan stops before reaching 100%.
• Cause: This can happen due to network interruptions or server-side processing timeouts.
• Fix:
  1. Read the error message displayed on the screen.
  2. Click Start a new scan.
  3. Try narrowing your filters to scan a smaller number of assets.
• Prevention: Avoid scanning the entire library at once if you have hundreds of thousands of assets; use filters to break the work into batches.

Issue: No Duplicates Found

• Symptoms: The results screen shows "No Duplicates Found" with a green checkmark.
• Cause: Your current filters are too restrictive, or your library is already clean.
• Fix: Click Start New Scan and remove some filters to broaden the search.