Slow "Verify" operation when copy with verification

[evgeniy]

Total commander version : 9.22a, 9.20, 64bit and 32bit
OS: windows 7
hardware: i5-2400, 16gb RAM

When I copy with verification directories with many big files, I found, than "Verify" part of operation works slow - about 110-114 MBytes/sec, but md5 calculation for same files on same drive works much faster (with full read speed of drive, sata HDD, up to 190 MBytes/sec).

[ghisler(Author)]

That's because "Verify" bypasses the disk cache, otherwise it would be pointless. md5 calculation uses the cache, so when you just copied a file and then calculate its md5, the data will be taken from the disk cache and no real write verification will be performed.

[DuBa]

Related question: file copy with verification using an external HDD is extremely slow (around 1MB/s, sometimes less). Would shifting to external SSD or NVME disk result in meaningful improvement? i.e. what is the bottleneck in file access when disk cache is bypassed, IOPS or something else?

[Stefan2]

Related question: file copy with verification using an external HDD is extremely slow (around 1MB/s, sometimes less). Would shifting to external SSD or NVME disk result in meaningful improvement? i.e. what is the bottleneck in file access when disk cache is bypassed, IOPS or something else?

Hello DuBa, welcome.

Which TC version and bitness do you use? On which OS?






 

[DuBa]

Hello DuBa, welcome.

Which TC version and bitness do you use? On which OS?

Thanks! I am using 64-bit version on Windows 10, behavior is the same on versions 10.0 and 11b5

[ghisler(Author)]

Would shifting to external SSD or NVME disk result in meaningful improvement?

Probably not. I would not use verify with external disks. Instead, copy the files normally, then disconnect the disk and reconnect it to flush the cache. Then just use the compare function in "synchronize dirs". This will me MUCH faster because it will use the disk cache, and still give you valid results because the cache was cleared. Unfortunately there doesn't seem to be a way to programmatically clear the cache.

[Dalai]

Related question: file copy with verification using an external HDD is extremely slow (around 1MB/s, sometimes less). Would shifting to external SSD or NVME disk result in meaningful improvement?

What kind of files do you copy? A speed of just 1 MB/s suggests that you copy a large amount of very small files. In such a case a solid-state disk would certainly improve the speed. Reaching such speeds when copying large files would definitely be not normal.

Regards
Dalai

[DuBa]

Thank you all! It is fascinating to me how many new ways to simplify life can be found in TC!

Probably not. I would not use verify with external disks. Instead, copy the files normally, then disconnect the disk and reconnect it to flush the cache. Then just use the compare function in "synchronize dirs". This will me MUCH faster because it will use the disk cache, and still give you valid results because the cache was cleared. Unfortunately there doesn't seem to be a way to programmatically clear the cache.

Fantastic, thank you! Compare by content function in "synchronize dirs" is identifying differences in files with identical names and size so I assume it is also using checksums?

I will use this method from now on but would it be possible to make "verify2" to do the same thing with one or two clicks or via options dialogue?

What kind of files do you copy? A speed of just 1 MB/s suggests that you copy a large amount of very small files. In such a case a solid-state disk would certainly improve the speed. Reaching such speeds when copying large files would definitely be not normal.

This is well known limitation of USB file transfer, and i am either copying a reasonable number of small(ish) files or a few .zips over 1GB. Verify was particularly slow for large files, I can do a test and load screenshots if anyone is interested.

Thanks again for help and fast feedback!

[ghisler(Author)]

Fantastic, thank you! Compare by content function in "synchronize dirs" is identifying differences in files with identical names and size so I assume it is also using checksums?

No, it is comparing the files byte by byte. Just make sure to check both "by content" and "ignore dates".