-[8.50b5] Verify, network and double reading of source (?)

[Sob]

This is an interesting one. Steps to reproduce:

1) Start clean-ini TC
2) F5 to copy some larger file from network drive (regular Windows share) to local drive
3) Check Verify (unchecked by default)
4) Start copy

Copy speed is now only half of what it could be. With fast ethernet (100Mbit) it shows only about 5MB/s and it really takes as much time as if it was 5MB/s. BUT incoming data on network adapter is double. I'm absolutely sure that nothing else uses the network except TC. And even byte counter in TCPView shows double amount of data for established SMB connection.

[HolgerK]

Not confirmed (monitoring the file operations while copying a small file using process monitor):

Testcase1: local drive -> Server/ Filesize is 10150 bytes:

09:45:11.6504802 TOTALCMD64.EXE 1472 ReadFile E:\test\Test.txt SUCCESS Offset: 0, Length: 10150, Priority: Normal
09:45:11.6505124 TOTALCMD64.EXE 1472 WriteFile \\<masked>\test\Test.txt SUCCESS Offset: 0, Length: 10150, Priority: Normal
09:45:11.6518494 TOTALCMD64.EXE 1472 ReadFile E:\test\Test.txt SUCCESS Offset: 0, Length: 10150, Priority: Normal
09:45:11.6577349 TOTALCMD64.EXE 1472 ReadFile \\<masked>\test\Test.txt SUCCESS Offset: 0, Length: 10150, I/O Flags: Non-cached, Priority: Normal

Testcase2: Server -> local drive / Filesize is 10150 bytes:

09:48:01.5554220 TOTALCMD64.EXE 1472 ReadFile \\<masked>\test\Test.txt SUCCESS Offset: 0, Length: 10150, Priority: Normal
09:48:01.5562046 TOTALCMD64.EXE 1472 WriteFile E:\test\Test.txt SUCCESS Offset: 0, Length: 10150, Priority: Normal
09:48:01.5625644 TOTALCMD64.EXE 1472 ReadFile \\<masked>\test\Test.txt SUCCESS Offset: 0, Length: 10150, Priority: Normal
09:48:01.5640857 TOTALCMD64.EXE 1472 ReadFile E:\test\Test.txt SUCCESS Offset: 0, Length: 10150, I/O Flags: Non-cached, Priority: Normal

Windows 7 / TC 8.50x64 b5 / 1Gbit LAN connection/ standard copy method.
Maybe another process (Virus scanner) scans the source file too?

And even byte counter in TCPView shows double amount of data for established SMB connection

The I/O flag for the read source operation is not set to "non-cached" (standard copy mode and small file)
This may be different for larger files and/or non - standard copy settings.

Regards
Holger

[Sob]

Are you sure about NOT confirmed? Because according to your logs:

ReadFile <source> - reading
WriteFile <target> - writing
ReadFile <source> - ???
ReadFile <target> - verification

[HolgerK]

"ReadFile <source> - ??? " I/O Flags: Non-cached is not set!
If the client OS does not cache the second read operation from network you may be right that the bytes are transfered twice from the server to the client.

In this case the verification would offer double security:
- The bytes are written correct to the local harddisk.
- The bytes are transfered correct via network from the server (at least no random transfer error).

Regards
Holger

[ghisler(Author)]

The problem occurs when you use the new default copy method, which uses CopyFileEx to copy. The problem with CopyFileEx is that there is no access to the actual file data. So TC starts a background thread and reads the file also in the background, for verification. This is done only when the file is > 10 MB. The two reads are synchronized, so Windows can take as much data as possible from the disk cache. This seems to work fine except when reading from network drives. In my tests, this is faster than copying the file, reading it again for verification, and then reading the target for verification.

If anyone has a better idea, please let me know.

[Sob]

Ok, lets call it feature. :)

I can't deny that double security as first side effect is not bad. But half speed as second side effect is too noticeable in this case. It's especially bad that it affects default copy method.
But there doesn't seem to be much what could be done, except for switching to other copy method for verified transfers from network. But if it would be the best idea (some other side effects perhaps?), I'm not sure.

[ghisler(Author)]

All you can do is use the old copy method, because it uses ReadFile/WriteFile - this allows TC to see and checksum the data while it is read. It has other downsides, like being slow when copying to USB sticks.