Could you add xxHash (xxh3, the fastest hash algorithm)?

Here you can propose new features, make suggestions etc.

Moderators: white, Hacker, petermad, Stefan2

Axis
Junior Member
Junior Member
Posts: 88
Joined: 2015-07-14, 18:53 UTC

Could you add xxHash (xxh3, the fastest hash algorithm)?

Post by *Axis »

Hi to all.

Please could you add xxHash (xxh3, the fastest hash algorithm: xxh3-avx2 is faster than system's RAM too) to TC's next version?

https://github.com/Cyan4973/xxHash/releases/tag/v0.8.0
https://github.com/Cyan4973/xxHash
https://github.com/Cyan4973/xxHash/wiki/Performance-comparison
https://github.com/Cyan4973/xxHash/wiki/Collision-ratio-comparison


Many thanks.
User avatar
vdijken
Member
Member
Posts: 181
Joined: 2016-07-30, 14:07 UTC
Location: The Netherlands

Re: Could you add xxHash (xxh3, the fastest hash algorithm)?

Post by *vdijken »

For what purpose?
gdpr deleted 6
Power Member
Power Member
Posts: 872
Joined: 2013-09-04, 14:07 UTC

Re: Could you add xxHash (xxh3, the fastest hash algorithm)?

Post by *gdpr deleted 6 »

2vdijken
Well, you should start with the reasonable assumption that Axis has a purpose/need/want for this particular hashing method, otherwise why would they ask for it?

Still, i wouldn't expect Christian to do this. There are quite a number of "exotic" hashing algorithms ("exotic" as in lacking widespread usage outside of narrow scenarios unrelated to PC mass storage such as communications/transmissions, low-perf embedded systems, etc.; and no, the mere existence of some package does not equate usage) that barely anybody uses. So what would be the upside of Christian spending the time and effort in implementing and maintaining an "exotic" hashing method in TC that quite likely barely any user will use in the foreseeable future?

I guess something "exotic" like this would be a case/suggestion for a packer plug-in (which then doesn't actually pack files, but rather generates a text file with the hash values of the files being "packed"), dependent on volunteer effort. If it ever happens in the future that xxHash will be on the verge of widespread popularity, it might then be the time to ponder whether it makes sense to include it into TC itself.
User avatar
nsp
Power Member
Power Member
Posts: 1803
Joined: 2005-12-04, 08:39 UTC
Location: Lyon (FRANCE)
Contact:

Re: Could you add xxHash (xxh3, the fastest hash algorithm)?

Post by *nsp »

To see how efficient it could be, you could try to build
a wdx to show hash in a column
a wcx to create a hash file (compress)
a wlx lister to verify hashed files

;)
Axis
Junior Member
Junior Member
Posts: 88
Joined: 2015-07-14, 18:53 UTC

Re: Could you add xxHash (xxh3, the fastest hash algorithm)?

Post by *Axis »

vdijken wrote: 2021-03-03, 10:05 UTC For what purpose?
Speed: It's the fastest non-cryptographic hash algorithm, working at RAM speed limit.
I really need speed for hashing very large data (GB's): to me It's more useful than many TC's Sha algorithm variants.

It's less and less "exotic :wink: ": go to the bottom of the web page https://cyan4973.github.io/xxHash/ (starting from "xxHash is used by").

Cheers
User avatar
solid
Power Member
Power Member
Posts: 747
Joined: 2004-08-09, 11:20 UTC

Re: Could you add xxHash (xxh3, the fastest hash algorithm)?

Post by *solid »

+1 for this request.
As plugin will also be fine.
User avatar
Hacker
Moderator
Moderator
Posts: 13052
Joined: 2003-02-06, 14:56 UTC
Location: Bratislava, Slovakia

Re: Could you add xxHash (xxh3, the fastest hash algorithm)?

Post by *Hacker »

Axis,
You could try to suggest adding xxHash to LotsOfHashes.

Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.
Axis
Junior Member
Junior Member
Posts: 88
Joined: 2015-07-14, 18:53 UTC

Re: Could you add xxHash (xxh3, the fastest hash algorithm)?

Post by *Axis »

Hacker wrote: 2021-03-03, 19:46 UTC Axis,
You could try to suggest adding xxHash to LotsOfHashes.
I think that's a dead project (2013) and imho tc's native integration, inside "Create Checksum File(s) (CRC32, MD5, SHA1)..." menu, is way better and really more useful than many SHA's.

Alternatively the cryptographic hash BLAKE3 would also be fine.


Please :wink: , Thanks

Cheers
Fla$her
Power Member
Power Member
Posts: 2244
Joined: 2020-01-18, 04:03 UTC

Re: Could you add xxHash (xxh3, the fastest hash algorithm)?

Post by *Fla$her »

Support++
Overquoting is evil! 👎
Faa
Junior Member
Junior Member
Posts: 5
Joined: 2013-10-06, 09:30 UTC

Re: Could you add xxHash (xxh3, the fastest hash algorithm)?

Post by *Faa »

My vote for xxHash. I tested xxHash today on my old laptop it's insanely fast. 😁
User avatar
nsp
Power Member
Power Member
Posts: 1803
Joined: 2005-12-04, 08:39 UTC
Location: Lyon (FRANCE)
Contact:

Re: Could you add xxHash (xxh3, the fastest hash algorithm)?

Post by *nsp »

Old topic but still good to revive it a little.
xxhash and specially xxhash3 64bit are really fast and could be internally used by TC to hash but also to replace md5 when looking for duplicates by content.
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Could you add xxHash (xxh3, the fastest hash algorithm)?

Post by *ghisler(Author) »

I have considered it, but there seem to be multiple incompatible versions of xxhash...
Author of Total Commander
https://www.ghisler.com
User avatar
nsp
Power Member
Power Member
Posts: 1803
Joined: 2005-12-04, 08:39 UTC
Location: Lyon (FRANCE)
Contact:

Re: Could you add xxHash (xxh3, the fastest hash algorithm)?

Post by *nsp »

ghisler(Author) wrote: 2022-03-18, 11:32 UTC I have considered it, but there seem to be multiple incompatible versions of xxhash...
you have multiple xxhash versions 32/64/128 and xxhash3 and some implementation are not purely following standards. But if you just consider xxhas3 64 to replace md5 calculation in duplicate finding (by content) and hash calculation after copy you have a benefit anyway. I use the command line version to hash big files and it is much faster than any md5sum i found.
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Could you add xxHash (xxh3, the fastest hash algorithm)?

Post by *ghisler(Author) »

The creation is not the problem - I could just create one version and be done. But then 1000 people complain that they wanted to check an xxhash created by some other tool and it fails because it's one of those many other variations.
Author of Total Commander
https://www.ghisler.com
User avatar
nsp
Power Member
Power Member
Posts: 1803
Joined: 2005-12-04, 08:39 UTC
Location: Lyon (FRANCE)
Contact:

Re: Could you add xxHash (xxh3, the fastest hash algorithm)?

Post by *nsp »

My suggestion is to use a modern efficient hash (xxhash3 or MetroHash) function to validate file operation or duplicate finder by content only...
But you are right as many implementation can use different seed to do the xxhash many tools with different result is not a "universal" hash for file exchange...
Post Reply