Slow big file copy, Speed Drops from 78->17 Mbyte

[komarto]

Hi,

When copying big file like 1080p HD movies which are from 8Gb to 40Gb,
From 2 different physical drives the speed at start is great 78-80MB
Then its starts dropping fast to about 17 which by then the memory of the
Machine is about 90% in use the system is lagging and non responsive.

I have tried using TC 7.04 and the betas of 7.5 i tried on vista 32bit and on se7en x64 RTM, i checked the copy modes in the settings menu "Big file" copy mode, always the same results starts fast then drops while grinding the system almost into complete HALT.

I even filled the text box which says what partition are on the same physical drive, still same result as the above.

The problem is with TC because in vista and in se7en when doing simple copy/paste the speed is maintained at about 65MB which is normal after the
First data burst. and the file copying is done fast.

I dont want to use the copy paste of windows i love my TC.

What else can i do to you guys solve this?

Thanks in advanced,
Komar

[petermad]

Have you tried either to chose "Use standard copy method (recommended)" or to choose "Use compatibility mode for the following drives" for the drive in question?

What buffer sizes do you use for big file copy mode?

[komarto]

Have you tried either to chose "Use standard copy method (recommended)" or to choose "Use compatibility mode for the following drives" for the drive in question?

What buffer sizes do you use for big file copy mode?

Yes i tried both methods.
The buffer size in big file copy mode is 10240k for same disk
and 64k for two physical drivs.

What does "compatibility mode" actually does?

What else can i try?

[ghisler(Author)]

Yes, try compatibility mode - it uses the Windows function CopyFileEx, which should behave similar to how the Explorer copies the files.

[komarto]

Hi,

As stated before i tried that and the result is the same.

only windows copy paste maintains the speed throughout the copy process
And does not hogs my pc.

What else can i do?

[ghisler(Author)]

Try using filemon to find out what is going on at the same time which slows down TC so much. You can then see for example whether a virus scanner is reading the just copied data or so.

[komarto]

Hi,

I used filemon (for win vista and se7en its process monitor).
I did a filter to show only the file I am copying (filter on path)
All i could see is TC copying reading 1 Mb and then writes chunck of 65k,
Am i right?

Any way no other interference was shown its just slowed down and chocked the system.

I did the same test with se7ven copy paste and it reads 1mb and writes 65k chunks, The same way that TC does BUT it does not chock the system...

Did you try copy a 10+ GB file drive to Drive on vista or Se7en?

[HolgerK]

2komarto
It must be somehow specific to your machine.
I'd just made a test (only 8Gbyte file, sorry the largest i've found here).

Configuration:
- Vista 32 SP2
- AMD-X2 Dualcore / 3GB Memory
- Copy between two different internal physical SATA hard disks / source disk=500GB(7200rpm) / destination disk=1TB(5400rpm) )
- file system: all NTFS
- Destination partition 100GB of 500GB free (last partition, so not the fastest region)
- TC7.50rc1 with [x] Use Standard copy method (recommended)
- Some background programs running (e.g. MediaCenter just recording from TV)
- Memory in use before test : about 1.3GB

As you said the copy process started with about 75 MB/s here too.
During the first seconds of the copy progress, the used memory increased until 1.7 GB is reached and keeped stable at this limit during the whole rest (80%) copy progress.
At the same time the copy speed slowed down a little bit until 50MB/s is reached for the rest of the time.
After the copy is finished the used memory dropped back to 1.3GB again.
CPU load: most of the time about 30% (mostly kernel time)

I've got the same duration, CPU load and memory consumption curve, if I copy with <Ctr+C><Ctrl+V> (Explorer Copy)

I can test this also later on Win7RC x64.

Do you see any significant difference from my configuration to yours?

Kind regards
Holger

[komarto]

Hi,

We basically the same setup
dual core amd
3gb mem
500Gb -> 1.5TB copy

I think you need to check >10GB files.

My problem happens with files over 6GB

I had this problem on vista 32bit and now with se7en x64.
I changed my hard drives and added memory.

I give up.

If windows copy worked well when i had vista and it works well when i have se7en x64, And i always had problem with copying big files using TC, I think the problem is with TC, This is what my logic tells me, I replaced everything from software to hardware and the problem persists.

I dont know what else to do

[Hacker]

Christian,
I still stand by my suggestion to create a small Tool to determine best copy method settings.

Roman

[HolgerK]

What is the best method?
The fastest?
The most reliable?
The resource-friendly?

Regards
Holger

[arkadian]

I have the same problem with TC. When copying big files, memory use is 90% making system slow and non-responsive.
The problem is not related to hardware, my machine is Core 2 Quad Extreme QX9650 with 6GB RAM.
Recently I was copying a huge 200GB file between two exactly same physical hard disk drives. Both Standard Copy Method and Big File Copy Mode cause the same phenomena.
I use TC 7.55a, however I had this issue with earlier versions as well.
It seems that Windows treats file copy encapsulated in TC as specific process and applies some specific memory support ultimately blocking the system.
Definitely this issue is a subject for further investigation.

[ghisler(Author)]

Total Commander 7.5x is using a standard Windows function (CopyFileEx) for copying. This function maps the file into memory, and it seems that Windows can use almost the entire memory for that purpose. To my knowledge, there is no way to limit this memory.

Total Commander does offer a special big file copy mode, where the disk cache is bypassed. You can enable it in Configuration - Options - Copy/Delete. For normal every day copying it's slower, though.

[arkadian]

I have googled for this topic and apparently the Author is right. When copying a file, Windows buffers it for future reads. While it works fine for relatively small and regular files, big files make system choking.
I assumed earlier that the problem lies within Total Commander (because I use only it for file management) and I was wrong.
Thus... Big File Copy Mode is a solution for copying big files :) However, I do not perform big file copy operations frequently, so I can live with switching this mode from time to time on.
And final advise for forum readers: do not use Big File Copy Mode on daily basis, read and write buffering implemented in Windows is very useful feature. It speeds up copy operations vastly. Big File Copy Mode is slower for small files, as the Author stated already.

[bobputnam]

You might see how Teracopy works in a similar situation. It's easy to integrate into TC too.