Index files, or find duplicates

Step 3 of the scan wizard, titled "How?", and the one that decides how long you wait. The subtitle says what the choice is: "Index your files for browsing, or also check them for exact duplicate copies."

Find duplicates is pre-selected and badged Recommended.

The two options

Both build the same list of files and folders, with the filter you chose on the previous step and the folder sizes that go with it. Find duplicates then goes on to compare the contents of files that could be copies of each other. That comparison is the part that takes the time.

What Find duplicates actually does

This is why even a 250 GB scan is quick: a file whose size is unique is ruled out immediately, without ever being opened. Most files on a normal drive are eliminated here, for free. Only files that share a size with something else go any further.

Those that do are compared in steps, and each step only looks at what the one before it could not separate — so the work shrinks as it goes:

Find duplicates runs all three, so a group it reports is an exact copy as a matter of fact, not as a strong indication. The earlier steps are not a shortcut you can stop at — they are how the last one is kept cheap.

That is the point the wizard's Why are big jobs fast? panel is making:

"Trawler reads the first and last block of each file before reading the rest — so a 100 GB scan finishes in seconds, not hours. …"

Your own times depend on the drive. A slow disk, a network location, or a CPU usage cap set in Settings → Scanning & performance all change the answer, and so does Efficiency mode, which is meant to.

Looking inside archives

Below the two options are two boxes, both cleared at the start of every scan:

With the first box ticked, Find duplicates also compares your files with the files stored inside every .zip and .7z in the scan, and inside every Solid archive Trawler made (the .tar.zst written when you archive with "Smallest file, one format"). A file that has an exact copy in an archive joins the same group on the Duplicates tab, and the archive copy is listed in that group's header. A .zip or .7z copy is compared byte for byte, the same as any other copy.

A Solid archive keeps its own record of every file it holds, written when Trawler made it, and that record is what the comparison uses, so the archive is not unpacked to find the copy. Before a copy in one counts as the copy you keep, Trawler unpacks it and checks it byte for byte. A copy that does not check out is dropped from the group.

Other archive types, such as .rar and .tar, are treated as ordinary files. So is a .tar.zst made by another program, a Solid archive made by a version of Trawler before this one, and an archive stored inside another archive.

Trawler never changes an archive. A copy inside an archive is shown, never ticked, and no action reaches inside it. You can still tick, move or remove the archive itself like any other file.

The second box can only be ticked when the first one is, and it decides whether an archived copy counts as a kept copy:

A copy in a protected folder is never ticked for you, whichever box you choose. If you tick the archive itself, its copies stop counting as kept, and the group goes back to keeping one loose copy.

Reading archives adds time after the comparison steps finish. Trawler only opens an archive member when its size matches a file that could be a copy.

When to pick Index files instead

Pick it when you want to look before you commit to anything:

• You want Folder sizes and Largest files to see where the space has gone, and you have no particular reason to think you have duplicates.
• The location is large or slow — a network share, an external drive — and you would rather get the list now and decide later.
• You only came for Similar files. Image, document, audio and video matching read the file list, not the duplicate results, so they work either way.

What you do not get is the Duplicates tab populated. Everything else in the results is the same.

When an Index files scan has no duplicate results, the result screen says so. Instead of what you can reclaim, it shows the total size of everything scanned, the headline "Here's what you scanned.", and a chart of it. Every file the scan listed is in the chart, broken down by type and then by family or extension. Re-indexing a session that has already found duplicates keeps those results, so its result screen still shows what you can reclaim. Your first scan, end to end describes the chart.

Running the check later

Nothing is wasted if you choose Index files and change your mind.

Open the Duplicates tab. It says "No duplicate check yet" over "These files are indexed. Run the duplicate check to find exact copies.", with a Find duplicates button. That works over the list you already have — your files are not walked again, and the filter you scanned with still applies.

The same is true in reverse: running a scan again keeps the work already done and re-reads only the files that have changed since.

Verify exact copies

On the Duplicates tab, Verify exact copies appears whenever the results came from sampling rather than from a full read. Its tooltip says what it is for:

"Matches so far were confirmed by sampling. Re-check the remaining groups against the full file contents. Your ticks are kept."

You will not normally see it, because Find duplicates already reads every byte of every possible match. It is there for results that came from somewhere else — an older saved session, for instance, opened in a newer version.

Changing the default

Settings → Scanning & performance → Default scan type decides which of the two the wizard pre-selects. It is set to Find duplicates.

Related

• While it's running — the step cards, and what a cancel keeps
• Reading the results — the four result tabs, and what each one shows
• Choosing what to remove — ticks, auto-select and protected paths
• Big scans — speed, memory, and network locations