Copy/Move function slow down?

[harrismail]

Copying or moving any files (but particularly large ones) start off quite fast and then seem to crawl along until the end.
The laptop is an Intel i7 with 16GB of RAM and is mostly idling?
I don't think it's TC at all but any thoughts?
Apologies for the poor screenshot.
Image: https://prnt.sc/9-dNVKY47z5t

[umbra]

Caching? When the operation starts, it writes data through a cache, limited only by the speed of the source disk. But when that fills up, the operation has to slow down to the actual speed of the target disk.

[Dalai]

This is normal behavior and is (mostly) independent from the used software to transfer files. Windows reads files into its file cache (in RAM) as fast as it can. It also writes them to their destination as fast as it can, but the destination is usually much slower (network, USB, VPN, etc). As soon as the Windows file cache is full, the read speed drops down to the write speed.

I said "mostly independent" because there are some tweaks that can be made (e.g. TC's big file copy mode), but nothing to magically increase the destination's bandwidth.

[Wilhelm M.]

... but nothing to magically increase the destination's bandwidth.

Even though TC is not a magical tool - is there a setting in the Configuration/Copy options that can reduce this speed reduction? There are many posts here in the forum that discuss these options, but I have never found out which are really good for copying large or (many) small files.

[Dalai]

2Wilhelm M.
The background transfer manager (BTM) with a reasonable speed limit is a good way to smooth out these speed differences and spikes. What is reasonable depends on the speed of the current transfer, i.e. source and target speeds. Within the same LAN (1 Gbit/s) a limit of 50-75 MB/s can make sense, but it also depends on how busy the target system is. Such a limit doesn't make much sense for a VPN or other internet connection though, unless you're lucky to have a GBit internet connection . USB drives may need a different limit again.

The BTM can also be set to automatically enable this transfer speed limit (see TC help, section 4.b).

[Wilhelm M.]

Well my internet connection has a TBit bandwidth, ob course Actually I meant a setting for local transfer of files, eg. to an external HD. I tested several settings but never experienced much differences. The process is always fast at the beginning, but rather snail-like at the end

[harrismail]

Thanks for all the advice. I completely understand now.

The screenshot I showed was moving a 3GB mkv film from the C drive on my WD PC SN740 Gen 4 NVMe to a (reasonably fast) USB thumb drive so I can see that the cache would struggle after a few seconds.

Obviously internal copying/moving on the Gen 4 NVMe drive is much faster.

Steve

[Dalai]

2Wilhelm M.
Try a rather small speed limit in BTM like 30-40 MB/s. Sure, this will reduce the transfer speed to USB 2.0 bandwidth, but you will see a difference and much smaller speed drops.

[petermad]

2Wilhelm M.
I tested copying an 11 GB file between two NTFS partitions on the same physical SATA hard disk.

Here is the Progress dialog with the standard copy method: https://tcmd.madsenworld.dk/standardcopymethod.png
And here it is for the Big file copy method (with default buffer setting 10240k): https://tcmd.madsenworld.dk/bigfilecopymethod.png

The speed is more steady with the big file copy method (around 50 M/s), but there are faster peaks with the standard method (10 M/s to 92 M/s).

I tried increasing the buffer in big copy mode to 102400k and it increased the general speed from 50 M/s to 60 M/s
I also tried increasing the buffer in big copy mode to 1024000k and it only increased the speed to about 74 M/s but it also made other processes on the system stop and caused big lags for other programs - so i can not recommend that! At test time I had 8 GB free physical memory, even with that 1 Gb buffer.

With the big file copy method an buffer size of 10240k copying the 11 GB file took 4:10 min
With the big file copy method an buffer size of 102400k copying the 11 GB file took 2:58 min
With the standard copy method copying the 11 GB file took 4:16 min (after a reboot)