Search Method
Comparison Criteria
You can search for duplicate files by a combination of the following criteria:- File Name
Finds files with the same name (without extension) - File Extension
Finds files with the same extension - File Size
Finds files with the same size - File Content
Finds files with the same content - Last Modified Date
Finds files with the same last modified date - Create Date
Finds files with the same create date - File Attributes
Finds files with the same attributes - Hard Link
Finds files with hardlinks - Similar Pictures
Finds similar or almost identical pictures
File Content
Compare method
- Byte by byte
- Checksum MD5 (128-Bit)
- Checksum SHA-1 (160-Bit)
- Checksum SHA-2 (256-Bit)
- Checksum SHA-2 (384-Bit)
- Checksum SHA-2 (512-Bit)
The checksum method reads the complete file content and creates a checksum from the content for further comparisons. Using a checksum could be much faster than a byte-by-byte comparison when you compare a lot of files. Comparing only the stored checksum will avoid to read the complete content of these files again and again. See Speed Comparison Test.
NOTE: Files with a size of 0 bytes will be automatically excluded from the search.
Ignore ID3 Tags of MP3 files
This option enables you to ignore the following tags inside MP3 files:Ignore Exif metadata of JPEG files
This option enables you to ignore the Exif metadata inside JPEG files.Ignore metadata of FLAC files
This option enables you to ignore all the metadata inside FLAC files.First compare a data block at the end of the files
With this option a data block at the end of both files will be compared first. The file comparison will be stopped if the data blocks are not identical. Otherwise AllDup continues the file comparison at the start of the file.Block size of the read buffer
This option enables you to change the block size of the read buffer for the content compare.- The file comparison starts with a block size specified by the start value.
- AllDup reads the block size from the two files into the memory and compares them.
- The file comparison will be stopped if the data blocks are not identical.
- Otherwise the block size will be increased by the increment value until it reaches the maximum value and continues comparing the files (2).
File Name
Compare all characters of a file name
Compare only characters at the beginning of a file name
This option enables you to compare only a certain amount of characters at the beginning of a file name and ignore the rest of the name.Ignore characters at the end of a file name
This option enables you to ignore a certain amount of characters at the end of a file name.
Partial name matching
This option enables you to find files with a similar name using the search criteria file name.Ignore specific characters from the file name
Here you can specify the characters to be ignored when using the search criteria file name. Example: Specifying the space character will be treat the file name "data 01" and "data01" as a duplicate.Options
Don't compare hard linked files
This option prevents the content comparison and the checksum creation of files which are hard links of themselves.Ignore time stamp
This option enables you to ignore the time stamp when using the search criteria create date or last modified date.Similar Pictures
This search method enables you to find similar or almost identical pictures between the image file formats BMP, JPEG, GIF, PNG and TIF. For each image will be a checksum calculated for further comparison with all other images.The following files will be automatically excluded from the search:
- Files without the extension BMP, JPG, JPEG, GIF, PNG and TIF
- Files with a size of 0 bytes
- Pictures with a width or height smaller than the specified compare size
- Corrupted, invalid or incomplete pictures (*)
- Files with a blocked read access (*)
Compare Method
-
aHash
This compare method creates a 64-bit checksum (8x8 pixel) or a 256-bit checksum (16x16 pixel) of an image. The checksum of aHash (Average Hash) is based on whether the pixel's value is greater than the average color of the image. -
dHash
This compare method creates a 64-bit checksum (8x8 pixel) or a 256-bit checksum (16x16 pixel) of an image. The checksum of dHash (Difference Hash) represent the relative change in brightness intensity. -
pHash
This compare method create a 64-bit checksum of an image. The checksum of pHash (Perceptual Hash) is based on the frequency spectrum data and a distance calculation doing a discrete cosine transform (DCT).
Percentage Match
Here you can specify the minimum percentage matching of two pictures. The calculated percentage matching between two pictures will be shown at the column Match. The percentage always refers to the reference picture of a group which will be shown in a different text color.Compare Size
Here you can specify the maximum width and height of the pictures to be compared. A lower compare size finds more similar pictures and speeds up the comparison time. A higher compare size finds more identical pictures and less similar pictures and of course needs more time to compare them.Recognition Rate
We have done several tests with the compare methods to determine the recognition rate with different image changes. For the test we have created several copies of a JPEG image (2304x3072px, 1.57MB) and changed the color, size or format of the copies. At all tests the original JPEG image and one altered copy of the image have been compared. The following table gives you an overview how good the compare methods recognize the changes at the image copies. A value of 100% means that the modified image copy will be seen as an full duplicate of the original image.| Modification of the image copy | Compare size |
aHash | dHash | pHash |
|---|---|---|---|---|
| Reduction of the JPG quality to 25% | 8x8 16x16 | 100% 100% | 98% 98% | 97% 100% |
| Reduction of the JPG quality to 50% | 8x8 16x16 | 98% 100% | 98% 98% | 97% 100% |
| Reduction of the JPG quality to 75% | 8x8 16x16 | 100% 100% | 98% 98% | 97% 100% |
| Conversion to grayscale | 8x8 16x16 | 98% 100% | 98% 99% | 97% 100% |
| Conversion to negative | 8x8 16x16 | 2% 0% | 3% 2% | 6% 6% |
| Reduction of the picture size by 50% | 8x8 16x16 | 83% 100% | 75% 97% | 53% 100% |
| Reduction of the picture size by 75% | 8x8 16x16 | 83% 100% | 73% 98% | 53% 100% |
| Enlargement of the picture size by 50% | 8x8 16x16 24x24 | 95% 95% | 92% 85% | 91% 88% 100% |
| Enlargement of the picture size by 100% | 8x8 16x16 24x24 | 94% 91% | 88% 64% | 91% 81% 97% |
| Enlargement of the picture size by 200% | 8x8 16x16 24x24 | 95% 91% | 84% 65% | 88% 78% 88% |
| Reduction of the picture height by 20% | 8x8 16x16 | 95% 100% | 92% 98% | 84% 100% |
| Reduction of the picture width by 20% | 8x8 16x16 | 97% 100% | 86% 98% | 72% 100% |
| Picture rotation by 90° to the left | 8x8 16x16 | 64% 56% | 58% 55% | 56% 53% |
| picture rotation by 90° to the right | 8x8 16x16 24x24 | 56% 56% | 50% 54% | 62% 50% 47% |
| Picture flipped vertically | 8x8 16x16 24x24 | 31% 38% | 59% 58% | 59% 56% 53% |
| Picture flipped horizontally | 8x8 16x16 | 89% 79% | 59% 55% | 56% 50% |
| Removed 100 pixel from the left edge | 8x8 16x16 | 97% 96% | 73% 80% | 75% 91% |
| Removed 100 pixel from the right edge | 8x8 16x16 | 100% 97% | 89% 80% | 97% 91% |
| Removed 100 pixel from the top edge | 8x8 16x16 | 94% 97% | 89% 88% | 84% 97% |
| Removed 100 pixel from the bottom edge | 8x8 16x16 | 100% 96% | 95% 90% | 100% 97% |
| Removed 100 pixel from the left, right, top and bottom edge | 8x8 16x16 | 95% 97% | 73% 79% | 84% 91% |
| Conversion to PNG 24bit | 8x8 16x16 | 100% 100% | 100% 100% | 100% 100% |
| Conversion to PNG 256 colors | 8x8 16x16 | 98% 100% | 98% 98% | 97% 100% |
| Conversion to PNG black/white | 8x8 16x16 | 97% 96% | 81% 70% | 88% 91% |
| Conversion to TIF | 8x8 16x16 | 100% 100% | 100% 100% | 100% 100% |
| Conversion to GIF 256 colors | 8x8 16x16 | 98% 100% | 98% 98% | 97% 100% |
| Conversion to GIF 128 colors | 8x8 16x16 | 100% 99% | 100% 98% | 97% 100% |
| Conversion to GIF 64 colors | 8x8 16x16 | 98% 99% | 97% 98% | 97% 100% |
| Conversion to GIF 32 colors | 8x8 16x16 | 98% 99% | 98% 95% | 97% 100% |
| Conversion to GIF 16 colors | 8x8 16x16 24x24 | 98% 100% | 98% 91% | 97% 97% 100% |
| Conversion to GIF 8 colors | 8x8 16x16 | 98% 100% | 94% 88% | 97% 97% |
| Conversion to GIF 4 colors | 8x8 16x16 | 98% 98% | 89% 84% | 91% 94% |
| Conversion to GIF black/white | 8x8 16x16 | 97% 96% | 81% 70% | 88% 91% |
| Conversion to BMP 24bit | 8x8 16x16 | 100% 100% | 100% 100% | 100% 100% |
| Conversion to BMP 256 colors | 8x8 16x16 | 98% 100% | 98% 98% | 97% 100% |
| Conversion to BMP 16 colors | 8x8 16x16 24x24 | 98% 100% | 98% 91% | 97% 97% 100% |