Could you add xxHash (xxh3, the fastest hash algorithm)?

[nsp]

@Hacker @ghisler(Author)
I personally have only concern about making verification and compare by content as fast and as reliable as possible. Anything that is 3 times faster than md5 or sha1 is a good solution for me

It would be awesome if we could have virtual tc wdx also exposing hash function

//

[ghisler(Author)]

Compare by content does not use hashes, it compares the two files directly.

I plan to use blake3 for the verify option when copying, and for the duplicate file finder.

[Fla$her]

FcHash xxhash3 - 12 sec
TC Blake3 - 11 sec

Something is doubtful that the implementation of the author of FcHash is of good quality.
Comparative indicators are too far from SMHasher, where:

Hash function	MiB/sec	cycl./hash	cycl./map	Quality problems
blake3_c	1285.91	340.01	552.63 (3)	no 32bit portability

Compare by content does not use hashes, it compares the two files directly.

But what about directory synchronization, which I wrote about here twice?

[nsp]

Compare by content does not use hashes, it compares the two files directly.

I plan to use blake3 for the verify option when copying, and for the duplicate file finder.

I was referring compare by content "for duplicate search" sorry for inaccuracy.

@Fla$her
About implementation in FcHash it is not as efficient as the reference one. Even if you take the reference code, compilation optimizations makes some differences and the xxhsum seems optimized. Also the times for smasher are for memory chunks which is in practical life, limited to read speed from HW.
On my machine, i have for 10Gib file on SSD hash calculation:

• ~3s for xxh3 almost the same for blake3 (rust) 12 threads

• ~8.5s blake3_c (1 thread)

• ~24s for md5

[ghisler(Author)]

But what about directory synchronization, which I wrote about here twice?

It also compares files directly. A direct comparison is much faster when the two files are big and differ not only near the end. It's also faster on SSDs and when the two files are on two different disks. A comparison via hashes is faster when most of the files are identical and are located on the same physical harddisk.

[hi5]

Just in case [... allow the user to add an "external checksum" tool ...] https://www.ghisler.ch/board/viewtopic.php?p=393767#p393767

[ghisler(Author)]

You can already add external checksum tools, either as a content plugin (LotsOfHashes), or as a packer plugin similar to DiskDirCrc.

[hi5]

I understand, but they are not "integrated" like the internal tools with the crc check window where it shows the results after you "verify" or just hit enter on the .sfv, .blake3 etc in the file panel:

test.blake3:
OK: test.txt

Errors: 0
OK: 1, not found: 0, read error: 0, wrong checksum: 0

So I thought a new type of plugin to pass/read results to that particular "window of TC".
(Not that I think you should spend much time on it as there are probably not too many people concerned about it)
Edit: and allow those "plugins" to be called from the create crc window just like you can now select a packer via ->
that way you don't have to add blake4,5,6 etc in the future to the gui yourself as others can simply update their "blake" plugin)

+1 for adding Blake3 btw.

[nsp]

You can already add external checksum tools, either as a content plugin (LotsOfHashes), or as a packer plugin similar to DiskDirCrc.

@ghisler(Author)
As you already embed a lot of hash calculation inside TC, could you expose those in new internal/virtual TC wdx fields ?
I like LotsOfHashes but it CRC/MD5/SHA/Blake could be visible directly with TC wdx it could be great !

@hi5
About external hash check, or hash plugin not sure if a lot of users will use it. In the meantime, you could make a WDX using command line tool or whatever and calculate or check on demand files. or a WLX on hash Files to report nicely check results.

[Fla$her]

On my machine, i have for 10Gib file on SSD hash calculation:

I checked folder on HDD with 40+ Gb files. Blake3 (TC) won (~23/27) in speed from xxhash3 (FcHash).

It also compares files directly. A direct comparison is much faster when the two files are big and differ not only near the end. It's also faster on SSDs and when the two files are on two different disks. A comparison via hashes is faster when most of the files are identical and are located on the same physical harddisk.

Thanks for the explanation.

[zhugecaomao]

@ghisler(Author)
As you already embed a lot of hash calculation inside TC, could you expose those in new internal/virtual TC wdx fields ?
I like LotsOfHashes but it CRC/MD5/SHA/Blake could be visible directly with TC wdx it could be great !

Support++++++, please consider to expose those in TC internal wdx.

I use the Custom Fields feature in the Copy/Move overwrite dialog and have set it to use the LotsOfHashes plugin. Very useful, because when an overwrite warning is displayed, the hash (in my case MD5 value) is also shown for both files, which may be the same, but only differ in date and time (occurs frequently when you have downloaded a file twice). but LotsOfHashes seems no more update & do not have BLAKE3 option.

[ghisler(Author)]

Hash values are not a good match for internal fields. Why? When a user adds a column with such a field and then sorts by that column, Total Commander has to load that field for all files before it can show the list. Fields which take a long time to load like hashes (when there are huge files) can delay the loading by a large time. I cannot prevent others from writing plugins which do this, but I prefer not to include it in the main program.

[zhugecaomao]

Hash values are not a good match for internal fields. Why? When a user adds a column with such a field and then sorts by that column, Total Commander has to load that field for all files before it can show the list. Fields which take a long time to load like hashes (when there are huge files) can delay the loading by a large time. I cannot prevent others from writing plugins which do this, but I prefer not to include it in the main program.

Thanks for your reply @ghisler.

I agree with you that adding column with hashes field may delay the file list loading, but indeed many users got this kind of requirement, not only for file list column, but also for Copy/Move overwrite dialog etc,. See LotsOfHashes wdx has 26000+ times download from totalcmd.net.

Users will know the impact of the hashes field (or maybe remind them when they trying to use it) to decide whether use it or not, but at least users should have the option to choose. thanks

[isidro]

It would be nice if you can also add the option to filter checksum verification by date (my latest verification has 35220 errors, most due to file updates). Showing file date/time and having a date filter would really help a lot.
Having a checkbox on COPY to generate chacksum hashes while copying would be useful, as asked here https://www.ghisler.ch/board/viewtopic.php?t=51773

[Usher]

There's already a checkbox for Verify there. I think that it would be better to chose verify method for that checkbox.