Add BLAKE2 to checksum methods

[ghisler(Author)]

Yes, it's maximum for SATA-III. But why would the CPU load be higher? It makes no sense, it's the same calculations just spread over more threads.

[lelik007]

2ghisler(Author)
Yes it’s about the maximum for this type of SSDs, but for cached reading multi-threaded BLAKE3 shows 3170 MiB/s as opposed to a single thread 1130 MiB/s with AVX only available in this CPU.

As I tested more extensively I found out multi-threaded BLAKE3 has peaks of CPU load to 80-90% sometimes, but the usual load for this particular PC is 30-40%. A single thread of BLAKE3 doesn’t have those peaks, I think this is how TBB works, maybe thread balancing.

[ghisler(Author)]

Yes, what happens is that instead of reading the big chunk of data linearly and calculating the checksum that data in a single thread, it starts multiple threads and passes separate chunks of the file file to each thread. The threads mostly wait for the data to be delivered, and then calculate their part of the checksum very quickly. In the rare cases where most of the threads get the data at the same time, the CPU load spikes to 80-90%, while normally the blocks come in at slightly different times. When you sum up the total CPU time used, it should not be significantly higher than the single thread function.

Anyway, I have now added code to distinguish between hard disks and SSDs. I'm using two different calls, one checks for the presence of the TRIM command (which hard disks don't have), and one checks for the flag "seek disadvantage" which HDDs report. If only one of them indicates a HDD, multi-threading is turned off an the file will be read linearly as in version 11.51 or older.

[lelik007]

2ghisler(Author)
Thank you, both things are good to have, multi-threading because we have more and more new PCs with any types of SSDs only. And HDD detection to avoid the slow reading speed caused by multiple random seek operations. As for CPU load I agree, I don't see it significantly higher overall. And anyway we have a fallback - CrcBlake3BlockSize=0.

[lelik007]

2ghisler(Author)
Thank you, you seems did it right, now my old HDD WD RE 1 TB gives 138 MiB/s (maximum) and I don't hear it spinning when reading a file which means it reads it linearly as the previous versions did. And my SATA-3 SSD gives 410 MiB/s (maximum) and I see multi-threading work of how the progress bar goes, the algo maps a file. So TC definitely can tell apart SSDs and HDDs and this really good!

TBH I don't have any NVMe SSD right now, but cached reading clearly shows multi-threaded BLAKE3 gives to 3200 MiB/s and this is maximum for this PC.
Single thread could only showed to 1200 MiB/s.

[ghisler(Author)]

Great, thanks for testing!

[MaxX]

one checks for the presence of the TRIM command (which hard disks don't have)

Some SMR HDDs have trim support to know if they have no need to read an existing stripe data before write operations.
You may get bad surprise here.

If anyone has such a SMR HDD, please check this.

[ghisler(Author)]

I check for both the presence of TRIM and the presence of the flag "does seeking have a time penalty". SMR HDDs should at least set that second flag.

[ghisler(Author)]

Warning: If you created BLAKE3 checksum files with RC2 64-bit, they may be invalid, if you created checksums for files smaller than 1 MB followed by files larger than 1 MB. You will have to create new checksum files with RC3 in this case.

The problem occurred because Total Commander didn't close the memory mapping to the previous file and used it for the one smaller than 1MB too.