MultiTheaded File Copy

[me.yay]

Lately i've seen that Robocopy allows to sepecify the number of parallel transfered files while copying/moving. ( /MT:N, where N is the number of threads)

This Feature speeds up filecopy, especially if lots of small files are involved.
Technicaly i can't explain the reason, but it does!

Instead of copying each file from the copy queue, it takes n-items from the queue and transfers them (almost) at once.

This Feature would be a real gain and makes tc even better for big copy Tasks than it allready is .

Regards,
Metin

[ghisler(Author)]

That doesn't make much sense to me, it would cause a lot of jumping of the harddisk heads between the different files. Or maybe you use an SSD (solid state disk)?

[krzychu]

Hi

It's a very old topic.

But multi-threaded copying is VERY useful as you're copying something from a corporate NAS over a VPN (Mapped drive). Such a NAS has a lot of performance, but due to the delays caused by the Internet and VPN when copying small files, the transfer is very very low.

The second thing SMB 3.0 has support for multithreading and it also helps when the NAS is connected via 2 x 1Gbps, and our computer 2.5Gbps. Then it is better to have more than one copying thread. But that's probably a separate topic. There are just so many uses when it will make things better (for example QoS on router).

This could work similarly to F2 Queue when copying e.g. F3 Multithread.

[ghisler(Author)]

But multi-threaded copying is VERY useful as you're copying something from a corporate NAS over a VPN (Mapped drive).

I'm already getting complaints from time to time that copying with TC saturates the connection and slows down the server. That would be even worse with parallel copying.

For SSDs it would be nice, but there doesn't seem to be an easy way to detect whether a drive is an SSD or HDD.

[hi5]

For SSDs it would be nice, but there doesn't seem to be an easy way to detect whether a drive is an SSD or HDD.

Perhaps consider a setting, similar to SamePartitions, and introduce AllowMultiThreadedDisks or something like that - when set, allow for an option to be chosen in the copy/move dialog e.g. Use MultiThreading (by default it remains off, but you can choose it for certain actions)

[krzychu]

I'm already getting complaints from time to time that copying with TC saturates the connection and slows down the server. That would be even worse with parallel copying.

Interesting complains Anyway, it could just be an option, disabled by default. It is important that it can work on network resources (without drive leather) and mapped drives. Now at work I got a proposal to use some scripts (Robocopy) for this, and I'd rather just use Total Commander which we paid for.

For SSDs it would be nice, but there doesn't seem to be an easy way to detect whether a drive is an SSD or HDD.

I guess for an SSD it's not a big deal. But if you're mainly working on NAS, with a lag between you and the NAS, it's very annoying to work with small files. Even deleting is slow (this could also be done multi-threaded).

[arko]

I support this idea. Make it optional and let users decide.

[krzychu]

Any news? Is there a chance for this option?

[Horst.Epp]

As a workaround, you can create TC buttons to use FastCopy.
It can run jobs in parallel and also has some settings to control how it's done.

[krzychu]

Source: Windows VPN and enterprise cached NAS
Target Local Intel M.2 SSD
Max download speed ~300Mbps
6088 files - 19,1MB (small files)
Total Commander: 39:27 (almost 40 minutes)
FastCopy (configured parallel, buffer 32MB): 07:11
Robocopy /E /MT:32: 00:29
Robocopy /E /MT:64: 00:24

Yes, Robocopy with 64 threads is almost 100 times faster than standard Total Commander copying. That's why I think it's a very useful thing.

[Ankha]

Now that SSD are widespread, I think it's a good option to have.

Also, it would be very useful on cloud drives such as Google Drive to use the whole bandwidth.

For SSDs it would be nice, but there doesn't seem to be an easy way to detect whether a drive is an SSD or HDD.

There are ways to figure it out, but it could simply be an option you can trigger on and off during copy (and it remembers the last choice based based on the source and destination drives for instance).

[hakito]

+1 for this feature. The problem why sequential file transfers are slow over VPN are because of TCP and SMB.

TCP has windowing (so also SFTP can be slow). And SMB wraps around file system operations.

So espescially on VPN connections it would make totally sense to copy files in parallel. My example:

* Server has uplink of 1 GBit
* Single file copy over wireguard around 10 MBit
* Every additional file copy also runs with around 10 MBit

There is no risk that additional copy streams would block the server for others. Also copying a single big files in chunks would be great!

[Tringi]

Regarding detection of SSDs there's DEVICE_SEEK_PENALTY_DESCRIPTOR used through DeviceIoControl API.
The result in .IncursSeekPenalty basically reports if the disk is SSD (false) or HDD (true).
I can provide sample snippet if you decide to implement it.