Add BLAKE2 to checksum methods

[ghisler(Author)]

Yes, it's maximum for SATA-III. But why would the CPU load be higher? It makes no sense, it's the same calculations just spread over more threads.

[lelik007]

2ghisler(Author)
Yes it’s about the maximum for this type of SSDs, but for cached reading multi-threaded BLAKE3 shows 3170 MiB/s as opposed to a single thread 1130 MiB/s with AVX only available in this CPU.

As I tested more extensively I found out multi-threaded BLAKE3 has peaks of CPU load to 80-90% sometimes, but the usual load for this particular PC is 30-40%. A single thread of BLAKE3 doesn’t have those peaks, I think this is how TBB works, maybe thread balancing.

[ghisler(Author)]

Yes, what happens is that instead of reading the big chunk of data linearly and calculating the checksum that data in a single thread, it starts multiple threads and passes separate chunks of the file file to each thread. The threads mostly wait for the data to be delivered, and then calculate their part of the checksum very quickly. In the rare cases where most of the threads get the data at the same time, the CPU load spikes to 80-90%, while normally the blocks come in at slightly different times. When you sum up the total CPU time used, it should not be significantly higher than the single thread function.

Anyway, I have now added code to distinguish between hard disks and SSDs. I'm using two different calls, one checks for the presence of the TRIM command (which hard disks don't have), and one checks for the flag "seek disadvantage" which HDDs report. If only one of them indicates a HDD, multi-threading is turned off an the file will be read linearly as in version 11.51 or older.

[lelik007]

2ghisler(Author)
Thank you, both things are good to have, multi-threading because we have more and more new PCs with any types of SSDs only. And HDD detection to avoid the slow reading speed caused by multiple random seek operations. As for CPU load I agree, I don't see it significantly higher overall. And anyway we have a fallback - CrcBlake3BlockSize=0.

[lelik007]

2ghisler(Author)
Thank you, you seems did it right, now my old HDD WD RE 1 TB gives 138 MiB/s (maximum) and I don't hear it spinning when reading a file which means it reads it linearly as the previous versions did. And my SATA-3 SSD gives 410 MiB/s (maximum) and I see multi-threading work of how the progress bar goes, the algo maps a file. So TC definitely can tell apart SSDs and HDDs and this really good!

TBH I don't have any NVMe SSD right now, but cached reading clearly shows multi-threaded BLAKE3 gives to 3200 MiB/s and this is maximum for this PC.
Single thread could only showed to 1200 MiB/s.

[ghisler(Author)]

Great, thanks for testing!

[MaxX]

one checks for the presence of the TRIM command (which hard disks don't have)

Some SMR HDDs have trim support to know if they have no need to read an existing stripe data before write operations.
You may get bad surprise here.

If anyone has such a SMR HDD, please check this.

[ghisler(Author)]

I check for both the presence of TRIM and the presence of the flag "does seeking have a time penalty". SMR HDDs should at least set that second flag.

[ghisler(Author)]

Warning: If you created BLAKE3 checksum files with RC2 64-bit, they may be invalid, if you created checksums for files smaller than 1 MB followed by files larger than 1 MB. You will have to create new checksum files with RC3 in this case.

The problem occurred because Total Commander didn't close the memory mapping to the previous file and used it for the one smaller than 1MB too.

[lelik007]

2ghisler(Author)
About this:

18.06.25 Fixed: Do not use multi-threading for blake3 checksums on systems with less than 8 logical processor cores, because it's not faster. Can be overridden via CrcBlake3BlockSize (64)

I disabled Intel Hyper-Threading Technology in BIOS for my old i7-2600K and It stayed with only 4 physical cores. Yes, in this case BLAKE3 isn't multi-threaded, but the issue is: then I did put CrcBlake3BlockSize=1024 in [Configuration] section of my wincmd.ini but it changed nothing, multi-threading for BLAKE3 doesn't work. It looks like TC ignores in this case CrcBlake3BlockSize=1024 for whatever reason.

[ghisler(Author)]

Which Windows version do you use? Blake3 multi-threaded needs Windows 7 or newer.
You can easily spot the multi-threaded function by looking at the percent bar - it updates far less frequently with multi-threading on.

[lelik007]

Which Windows version do you use? Blake3 multi-threaded needs Windows 7 or newer.
You can easily spot the multi-threaded function by looking at the percent bar - it updates far less frequently with multi-threading on.

Though the hardware is old it’s Windows 10 22H2 x64 there, and a progress bar looks exactly like single-threaded BLAKE3, compared to TC 11.55 RC6 where multi-threading is default and it works.

From my perspective it looks like TC favors its internal setting defined for:

Do not use multi-threading for blake3 checksums on systems with less than 8 logical processor cores.

And ignores explicit usage of CrcBlake3BlockSize=1024 in wincmd.ini

[ghisler(Author)]

No, the code looks like this:

Code: Select all

  if SysInfo.dwNumberOfProcessors<8 then
    CrcBlake3BlockSize:=0
  else
    CrcBlake3BlockSize:=1024;
  CrcBlake3BlockSize:=LoadTcSectionInt(captconfw,pr_CrcBlake3BlockSize,CrcBlake3BlockSize);
  if CrcBlake3BlockSize>2048 then
    CrcBlake3BlockSize:=2048*1024*1024   // Warning: values>3344 give invalid results!
  else if CrcBlake3BlockSize<0 then
    CrcBlake3BlockSize:=0
  else
    CrcBlake3BlockSize:=CrcBlake3BlockSize*1024*1024;

I first set the default value depending on the number of processors. Then I call the function to read from the ini file. Finally I convert the value to Megabytes.

The checks I do after that can't be overridden, though:
1.Windows 7 or newer
2. File larger than 1 MByte
3. File doesn't have FILE_ATTRIBUTE_SPARSE_FILE attribute
4. File is located on SSD

[lelik007]

2ghisler(Author)
I understood, initially I did my tests on a wrong drive which is HHD, so this limitation took place:

4. File is located on SSD

So CrcBlake3BlockSize=1024 works.

I redid the tests.
And even with 4 physical core old CPU like i7-2600K and SATA-III SSD multi-threaded BLAKE3 is faster than single-threaded, maybe it's not like when using the modern hardware, but 2 times for sure.

[ghisler(Author)]

The i7-2600K supports hyperthreading, so it has 8 logical cores according to Windows GetSystemInfo, and Blake3 multithreading should be on by default.