Slow "Verify" operation when copy with verification

Please report only one bug per message!

Moderators: white, Hacker, petermad, Stefan2

Post Reply
evgeniy
New Member
New Member
Posts: 1
Joined: 2019-08-26, 00:34 UTC

Slow "Verify" operation when copy with verification

Post by *evgeniy »

Total commander version : 9.22a, 9.20, 64bit and 32bit
OS: windows 7
hardware: i5-2400, 16gb RAM

When I copy with verification directories with many big files, I found, than "Verify" part of operation works slow - about 110-114 MBytes/sec, but md5 calculation for same files on same drive works much faster (with full read speed of drive, sata HDD, up to 190 MBytes/sec).
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Slow "Verify" operation when copy with verification

Post by *ghisler(Author) »

That's because "Verify" bypasses the disk cache, otherwise it would be pointless. md5 calculation uses the cache, so when you just copied a file and then calculate its md5, the data will be taken from the disk cache and no real write verification will be performed.
Author of Total Commander
https://www.ghisler.com
DuBa
Junior Member
Junior Member
Posts: 3
Joined: 2023-06-06, 10:14 UTC

Re: Slow "Verify" operation when copy with verification

Post by *DuBa »

Related question: file copy with verification using an external HDD is extremely slow (around 1MB/s, sometimes less). Would shifting to external SSD or NVME disk result in meaningful improvement? i.e. what is the bottleneck in file access when disk cache is bypassed, IOPS or something else?
User avatar
Stefan2
Power Member
Power Member
Posts: 4132
Joined: 2007-09-13, 22:20 UTC
Location: Europa

Re: Slow "Verify" operation when copy with verification

Post by *Stefan2 »

DuBa wrote: 2023-06-06, 10:19 UTC Related question: file copy with verification using an external HDD is extremely slow (around 1MB/s, sometimes less). Would shifting to external SSD or NVME disk result in meaningful improvement? i.e. what is the bottleneck in file access when disk cache is bypassed, IOPS or something else?
Hello DuBa, welcome.

Which TC version and bitness do you use? On which OS?






 
DuBa
Junior Member
Junior Member
Posts: 3
Joined: 2023-06-06, 10:14 UTC

Re: Slow "Verify" operation when copy with verification

Post by *DuBa »

Stefan2 wrote: 2023-06-06, 11:00 UTC Hello DuBa, welcome.

Which TC version and bitness do you use? On which OS?
Thanks! I am using 64-bit version on Windows 10, behavior is the same on versions 10.0 and 11b5
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Slow "Verify" operation when copy with verification

Post by *ghisler(Author) »

Would shifting to external SSD or NVME disk result in meaningful improvement?
Probably not. I would not use verify with external disks. Instead, copy the files normally, then disconnect the disk and reconnect it to flush the cache. Then just use the compare function in "synchronize dirs". This will me MUCH faster because it will use the disk cache, and still give you valid results because the cache was cleared. Unfortunately there doesn't seem to be a way to programmatically clear the cache.
Author of Total Commander
https://www.ghisler.com
User avatar
Dalai
Power Member
Power Member
Posts: 9364
Joined: 2005-01-28, 22:17 UTC
Location: Meiningen (Südthüringen)

Re: Slow "Verify" operation when copy with verification

Post by *Dalai »

DuBa wrote: 2023-06-06, 10:19 UTCRelated question: file copy with verification using an external HDD is extremely slow (around 1MB/s, sometimes less). Would shifting to external SSD or NVME disk result in meaningful improvement?
What kind of files do you copy? A speed of just 1 MB/s suggests that you copy a large amount of very small files. In such a case a solid-state disk would certainly improve the speed. Reaching such speeds when copying large files would definitely be not normal.

Regards
Dalai
#101164 Personal licence
Ryzen 5 2600, 16 GiB RAM, ASUS Prime X370-A, Win7 x64

Plugins: Services2, Startups, CertificateInfo, SignatureInfo, LineBreakInfo - Download-Mirror
DuBa
Junior Member
Junior Member
Posts: 3
Joined: 2023-06-06, 10:14 UTC

Re: Slow "Verify" operation when copy with verification

Post by *DuBa »

Thank you all! It is fascinating to me how many new ways to simplify life can be found in TC! :D
ghisler(Author) wrote: 2023-06-06, 13:42 UTC Probably not. I would not use verify with external disks. Instead, copy the files normally, then disconnect the disk and reconnect it to flush the cache. Then just use the compare function in "synchronize dirs". This will me MUCH faster because it will use the disk cache, and still give you valid results because the cache was cleared. Unfortunately there doesn't seem to be a way to programmatically clear the cache.
Fantastic, thank you! Compare by content function in "synchronize dirs" is identifying differences in files with identical names and size so I assume it is also using checksums?

I will use this method from now on but would it be possible to make "verify2" to do the same thing with one or two clicks or via options dialogue?
Dalai wrote: 2023-06-06, 14:51 UTC What kind of files do you copy? A speed of just 1 MB/s suggests that you copy a large amount of very small files. In such a case a solid-state disk would certainly improve the speed. Reaching such speeds when copying large files would definitely be not normal.
This is well known limitation of USB file transfer, and i am either copying a reasonable number of small(ish) files or a few .zips over 1GB. Verify was particularly slow for large files, I can do a test and load screenshots if anyone is interested.

Thanks again for help and fast feedback!
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Slow "Verify" operation when copy with verification

Post by *ghisler(Author) »

Fantastic, thank you! Compare by content function in "synchronize dirs" is identifying differences in files with identical names and size so I assume it is also using checksums?
No, it is comparing the files byte by byte. Just make sure to check both "by content" and "ignore dates".
Author of Total Commander
https://www.ghisler.com
Post Reply