[Implemented] Use the new CRC32 to allow users to optionally speed up the Verify after copying?

[sky66]

Referring to this :
viewtopic.php?t=84116

After testing, it is confirmed that the time to create and verify CRC32 checksum is significantly reduced, while the speed for MD5 remains the same.

In the past, because the Verify (MD5) after copying was slow, I didn’t enable it when dealing with many or large files.
I’d like to suggest: is it possible to let users configure the ini file to change the default Verify method after copying from MD5 to CRC32 to speed up the Verify process?

[ghisler(Author)]

I think that MD5 is inappropriate for this because it's too short. These days, files are often larger than 2^32 bytes. But I could use Blake3 instead of MD5 automatically, as i do now in the duplicate file finder.

[lelik007]

2sky66
It's not a good idea to use CRC32 for file verifying it's not able to resist collision these days, because we have too many files for 32 bit length. It's even less appropriate for this than MD5. I actually meant the old .sfv files on the modern flash memory based drives when I did this suggestion. libdeflate's Slicing-by-8 (though it doesn't rely on SIMDs) is ~ 1100-1200 Mbyte/s, BLAKE3 has about the same speed with SSE 4.1 and it's 1800 Mbyte/s with AVX2,
I don't have AVX512 to test BLAKE3 with this set. MD5 is ~ 500 Mbyte/s, the difference is obvious.

[browny]

It's not a good idea to use CRC32 for file verifying it's not able to resist collision these days

This is not a file verification.
The goal is to verify if source data that was transferred and written to the destination, could be read correctly. What would be the probability of random errors in read/transfer/write/read sequence to create a collision for CRC32?

[lelik007]

2browny
Do you mean that BLAKE3 is less appropriate for this than CRC32?

[browny]

A program that uses AVX2 simply fails to start in Windows XP (Virtualbox VM).

[ghisler(Author)]

A program that uses AVX2 simply fails to start in Windows XP (Virtualbox VM).

Do you mean that you can't use Blake3 in Virtualbox? That would be odd, because Blake3 only uses AVX2 if the processor reports that it's available.
I have just tried it, and Blake3 checksums can be created both in Windows XP and Windows 10 running in Virtualbox.

[lelik007]

2ghisler(Author)
Christian, browny claimed I had no knowledge, he's exactly right because I don't understand how this:

A program that uses AVX2 simply fails to start in Windows XP (Virtualbox VM).

can affect x64 version in any way and why It should shares the same as x32 version.
So my opinion is simple: for x64 it should be BLAKE3 and for x32 I don't care at all.

[browny]

Do you mean that you can't use Blake3 in Virtualbox? That would be odd, because Blake3 only uses AVX2 if the processor reports that it's available.

It means AVX2 could not be used in hash calculations, and tests show that SSE2 implementation is over 4 times slower.
Cryptogrhapic hash is an onverkill for basic data verification; maybe even simple checksum using combination of rotate and xor might be good enough to catch disk or memory errors. Archivers keep using CRC32, and nobody is afraid of large file sizes.
Or take a look at xxHash for fast non-cryptographic hashing.

[sky66]

25.11.24 Added: F5 Copy with verify: Use Blake3 instead of MD5 hashes if available to check whether source and target are the same (faster, double hash length) (32/64)

After testing, the 11.50b8 version with Blake3 is significantly faster than the 11.50b7 version. Thank you.

[sky66]

In the Help documentation, the "F5: Copy" section under File operations still only mentions MD5.
"With the option Verify enabled, Total Commander reads the copied file again after copying finishes, and compares its MD5 checksum with the original. The disk cache will be bypassed."

If it could be changed to MD5/BLAKE3 like in the "Configuration - Copy/Delete" section, it would be easier to understand.

Also, I tested it on three different computers, and the speed has noticeably improved.
I suggest considering highlighting in the "What's new in this version?" section under "File operations" that version 11.50 has improved the verify speed. After all, file copying is an important action in file management.

[ghisler(Author)]

In the Help documentation, the "F5: Copy" section under File operations still only mentions MD5.

Confirmed, I will change it.

I suggest considering highlighting in the "What's new in this version?" section

I didn't put it there because the main limiting factor with most disks (at least SATA disks and SSDs) is the raw reading speed.

[j7n]

That is only if you couple it with a brand new processor. People have older processors or the lower end Celeron/Pentium as processors depreciate in value rapidly, and have diminishing returns for more money paid.

For me hashing times in TC 11.02 are the following: ‎BLAKE < SHA1 = MD5 < CRC32 << SHA256. I just timed it with a clock. A second run is likely from the cache, so you can simply do the operation twice and compare the speeds. The speeds have gone up considerably from what I remember in older software like WinHex.

I don't think you can catch every error as there bound to be collisions, but have a high enough confidence.

[ghisler(Author)]

2j7n
Which processor?

[lelik007]

2j7n
This new for TC CRC32 method mentioned by sky66 is only implemented since v11.50 beta 7, you can't check it anyhow in 11.0x even with a disk cache.