Add BLAKE2 to checksum methods

[lelik007]

2ghisler(Author)
Yes, I disabled Hyper-Threading on purpose, I wanted to check if we're able to forcefully enable multi-threading with CrcBlake3BlockSize=1024 in wincmd.ini, so yes, we're able to do this.

And as for me I'd like when I have more time to play more with the buffer size.

[ghisler(Author)]

If you want to test without the disk cache, you can use this trick:
1. Go to Configuration - Option - Copy/Delete
2. Set the copy mode to "Also use big file copy mode"
3. Start copying the file you want to test with F5
4. Immediately cancel copying
This mode opens the file in mode "no buffering", which has the side effect of removing the file from the disk cache.
After doing this, the Blake3 function will read the file from disk, which is considerably slower than reading it from memory.

[lelik007]

2ghisler(Author)
I understood, and actually I forgot to mention that with i-2600K HT TC uses multi-threading by default.
But I meant BLAKE3 buffer, which is 1024 Megabytes in this case, but it's more for the researching purposes than for practical,
I think 1024 Megabytes is OK.

[white]

2ghisler(Author)
I did some testing now:

Code: Select all

Blake3 hash creation with TC 11.55rc7 64-bit

Hardware:
	CPU i7-8700
	32GB memory
	NVMe SSD

Tested with 7 applications open, 24,5GB memory available

Testfile: 
	4GB Windows 10 iso file 

Timings (no cache/cache):
	default (CrcBlake3BlockSize=1024): 5s / 1s
	CrcBlake3BlockSize=0:              2s / 2s
	CrcBlake3BlockSize=1:              6s / 3s
	CrcBlake3BlockSize=2:              5s / 3s
	CrcBlake3BlockSize=2048            error reading the file

Testfile: 
	12GB file (containing 3 copies of previous test file)

Timings (no cache/cache):
	default (CrcBlake3BlockSize=1024): 14s / 3s
	CrcBlake3BlockSize=0:               8s / 6s
	CrcBlake3BlockSize=1:              17s / 9s
	CrcBlake3BlockSize=2:              20s / 6s
	CrcBlake3BlockSize=2048:           error reading the file

[ghisler(Author)]

Interesting results - so the multi-threaded function is always a lot faster when cached, but your SSD is somehow slowing it all down, as if it has a high latency when switching between different blocks compared to serial reading. On my own system, the multi-threaded function is about 2-3 times faster even uncached on NVMe, and has the exact same speed on SATA SSDs.

In conclusion, someone who uses this function a lot will have to do their own benchmarks because every system is different.

[lelik007]

2white

CrcBlake3BlockSize=2048

2048 is the limit. 2047 is the maximum value you can use.
If you have some time and patience you could test how the BLAKE3 buffer size plays it's role, like:
CrcBlake3BlockSize=256
CrcBlake3BlockSize=512
CrcBlake3BlockSize=1536
I mean will it be faster, slower or just the same.

2ghisler(Author)

In conclusion, someone who uses this function a lot will have to do their own benchmarks because every system is different.

So true, because right now I can't tell apart when multi-threaded BLAKE3 is faster on NVMe or when it's slower.

[DontUseMe]

I am dealing with lot of big files and like to create blake3 chksum files.
those big-Giles ( 10+GB up to 50GB) are on SATA(mechanical drives, USB Drives or samba-shares.

The fastest option for me ist right now to stick with 11.51 ... and to use the updated 7z/rar - DLLs.
https://www.totalcommander.ch/beta/dllupdate1155.zip ( thanks for this package )

I mixed the dlls in both ways .
Newer "Blake3.DLL" into 11.51 or older "Blake3.DLL" into 11.56.


"CrcBlake3BlockSize=1" helped a a lot with 11.56 creating blake3 sums on SATA-HDDs ( max 145MB/s) but ... in the
end 11.51 with the old Blake3-DLL seems to be the fastest for me cause my big files are usually not on an ssd.
( constant 145MB/s on a SATA-HDD donig 145MB/s ...usually )

11.56 with "CrcBlake3BlockSize=1" is slightly slower ...
( less than 120-140MB/s ..not constand speed... a SATA-HDD donig 145MB/s ...usually )

and really bad without "CrcBlake3BlockSize=1" ( less than 30MB/s on a SATA-HDD donig 145MB/s ...usually )

So nothing so solve ... just a story

btw ... another little thing:
The Blake3 creation is slow in responding if you want to abort the creation in 11.56
With 11.51 the the abort of a blake3-creation is instant.

[ghisler(Author)]

That's odd, TC 11.55/11.56 should not use multi-threading on mechanical drives or USB drives because it isn't faster - it should get the same speed as the one from 11.51. Is your HDD some kind of hybrid hdd/ssd combo?

[lelik007]

2DontUseMe
You can check what CrcBlake3BlockSize=0 does, for some people it gives the best results.

[DontUseMe]

"Is your HDD some kind of hybrid hdd/ssd combo?"
No.

MB Gigabyte Z390 / Intel 9700 ... internal SATA(AHCI) Drive WD Purple 4TB or "WD43PURZ" ...
classic CMR Disk as AHCI Drive ...Win11 24H2 Pro .. no Fancy Stuff.
( same thing with "modern" MBs like B760M or Z790 )

Blue TCMD 11.51 constant 145 MB/s
https://i.postimg.cc/RhwJnbPS/TCMD-11-51-blake3-create.png

Orange TCMD 11.56 ( No CrcBlake3BlockSize in INI File ) constant 25 MB/s
https://i.postimg.cc/sXkZxR4H/TCMD-11-56-blake3-create.png

But like i said ... no need to fix anything ... not for me.
TCMD 11.51 with the updated 7z/Winrar DLLs ist fine for me.

BUT .. if you want me to test something for you ... you can contact me with email.
Last email you wrote (replied ) to me was:
27.10.2022, 12:25
Subject "Re: TCMD 10.5x und FTP"

I dont want to post my email here

[ghisler(Author)]

MB Gigabyte Z390 / Intel 9700 ... internal SATA(AHCI) Drive WD Purple 4TB or "WD43PURZ" ...
classic CMR Disk as AHCI Drive ...Win11 24H2 Pro .. no Fancy Stuff.

Strange, I'm, using two functions to detect SSDs, and only use multi-threading if both return true. Somehow your HDD is detected as an SSD.

[lelik007]

2ghisler(Author)
Hello!
If you're updating every component of TC for v11.57 as I see in the history, look at this release:
https://github.com/BLAKE3-team/BLAKE3/releases/tag/1.8.3
It's relatively new.

[ghisler(Author)]

I'm currently on 1.8.2 and compared the sources with 1.8.3, but there aren't any significant code changes between the two in the C implementation. You can check this yourself via synchronize dirs with options "ignore date" and "by content".

[lelik007]

2ghisler(Author)
I didn't mean BLAKE3 v1.8.3 specifically has the significant changes.

I meant that there are many internal components TC contains, like BLAKE3, libdeflate, brotli and all of them were updated after the latest TC release - TC v11.56, the beta test is a good occasion to update them.

I reminded just in case you missed the updates of any of these components.

[ghisler(Author)]

I updated the ones where it matters, e.g. important bug fixes like with 7z and WinRAR.