[8.0b13x64] bug in synchronize dirs

[fg_2002fr]

on Windows 7 professional 64bits with 4GB memory.

I needed to synchronize 2 dirs on the same hard disk with 200 files. I deleted the 150 identical files in the right pane then all cores of my quad cores CPU showed 100% activity, and memory consumption increased indefinitely, even after all files have been deleted, and even after the synchronize window was closed.
Memory consumption rose from 10Mo to 1,5Go before I closed TC.

[ghisler(Author)]

This us unfortunately a Windows bug: Windows creates a new thread for each file it deletes to recycle bin. I cannot use the function which deletes multiple files in one step, because when there are files from multiple directories, Windows deletes the entire directories, and not just the selected files...

The only way is to not use recycle bin (hold down Shift when starting the operation).

[ghisler(Author)]

This should be much faster now in beta 14, because I'm now deleting one directory at a time, not one file at a time.

[ghisler(Author)]

Could anyone try this, please?

[karlchen]

Hello, Christian.

I can partially reproduce the reported behaviour using
+ Total Commander 8.0ß13 64-bit on
+ VistaDelete=1 used
+ Server 2008 R2
+ having 2 CPU kernels and 16 GB of RAM

Testing foldertree:
+ Oracle 11g client setup extracted to disk having
+ having 721 subfolders and 2731 files

Put the this foldertree on two different partitions (two different drives). Removed a small number of files and folders from the left partition.
Launched T.C. synchronize directories, configured it to show equal folders existing on both sides only.
Marked all found 2709 files (ctrl-a), selected "delete right".
Monitored the resource usage with the help of Process Hacker 2.27 64-bit.

Result:
Problem partially reproduced. Both CPU kernels jumped up to 100% usage and stayed there for more than a minute till the last file had been sent to the recycle bin.
Memory consumption did not increase in any excessive way, maybe 16 GB of RAM are too much or vice versa 2709 file were not enough.

Repeated the test this time using Total Commander 8.0ß22 64-bit.

Result:
Both CPU kernels jumped up to almost 100% usage. CPU usage went up and down a bit during the delete operation. Deleting took about 1/3 of the time which T.C. 8.0ß13 had taken.

There had not been any extraordinary RAM usage under T.C. 8.0ß13, T.C. 8.0ß22 did not use an extraordinary amount of RAM either.

Though I could not reproduce the enormous RAM usage, I can confirm that deleting a large number of files on one side in the synchronize directories panel is much faster on T.C. 8.0ß22 64-bit than it was on T.C. 8.0ß13 64-bit and that it stresses the CPU less.

Kind regards,
Karl
--
Environment:
+ Total Commander 8.0ß13 64-bit (VistaDelete=1 used) and
+ Total Commander 8.0ß22 64-bit (VistaDelete=1 used) on
+ Server 2008 R2
+ having 2 CPU kernels and 16 GB of RAM

[ghisler(Author)]

Thanks for testing! With 2709 files in 721 subfolders, you get about 3-4 files per folder to delete - so it should be about 3-4 times faster than when deleting each file separately.

[karlchen]

Hello, Christian.

The tests were done on a server machine. Some background activities which may slow down a particular action or may not do so will always be there and exercise some influence on such tests, even on a Friday afternoon.

Anyway, because I had not monitored the exact figures for RAM usage, I repeated the same tests, just the other way round: T.C. 8.0ß22 64-bit was first and T.C. 8.0ß13 64-bit second to perform the delete operation.

RAM usage:
Neither of the two increased its private bytes above 25 MB (according to Process Hacker).
So, still no way of reproducing excessive RAM usage during delete operations in this environment. Which means: initial problem report still not fully reproduced.

CPU usage:
Same as last time. Constant 100% for T.C. 8.0ß13 64-bit during the whole delete operation.
Peaks of 100% CPU usage for T.C. 8.0ß22 64-bit.

Delete duration:
Again T.C. 8.0ß22 64-bit deleted the files and folders in a little more than 30 seconds.
This time it took T.C. 8.0ß13 64-bit almost 4 minutes to complete its delete job. (Yet, some of the additional timespan, compared to the first test, may have been caused by some other activity on the server which was unrelated to my T.C. experiments.)

Kind regards,
Karl

[umbra]

Hi, I have made a few tests of my own.
Basically, I observed everything mentioned in the first post, but not with 150 files. I needed to delete at least a thousand files to see it. Note: when the delete dialog says, that the operation is finished and closes itself, the operation still continues (consumes memory, cpu, disk) for a while. The more files one deletes, the longer it continues. I think that's what fg_2002fr thought by "memory consumption increased indefinitely, even after all files have been deleted".

I tested TC8 beta 13 and final (both 32b and 64b versions) with enabled VistaDelete.

TEST 1 The test folder contained 7500 files scattered in several nested directories (250 files per folder).
Results:
There was almost no difference between 32b and 64b versions. The latter one was usually a bit faster, but only marginally.

TC8 beta 13:
The operation took 35 - 40 minutes and memory consumption went up to 1.5 - 1.6 GiB.

TC8 final:
The operation took 0.5 - 1 minute and memory consumption went up to 30 - 40 MiB.

TEST 2 The test folder contained 5000 files in a single directory.
Results:
Again, there was almost no difference between 32b and 64b versions.

TC8 beta 13:
The operation took 15 - 20 minutes and memory consumption went up to 1 GiB.

TC8 final:
The operation took 20 seconds and memory consumption went up to 70 MiB.

So in my test, the removal of nested directories was 70-times faster and required 50-times less memory. And the removal of files, that were all in the same directory was 50-times faster and required 15-times less memory. I call that a massive improvement.

[ghisler(Author)]

Unfortunately that's the best I could do - when I pass files from multiple directories to the vista delete function, the function starts delting the complete directories, not just the files inside...

[derdiedas]

If I delete many files TC unfortunately still takes 100% CPU on Win7/64bit.
It also takes all the available RAM up to the point where it stops working.
It seems the amount of RAM relates to the amount of data to be deleted. So if I try to delete more GB than I have RAM it will not work.
Please tell me if I can provide any log files.

Code: Select all

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Application Error" /> 
  <EventID Qualifiers="0">1000</EventID> 
  <Level>2</Level> 
  <Task>100</Task> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2012-10-26T12:26:49.000000000Z" /> 
  <EventRecordID>1586</EventRecordID> 
  <Channel>Application</Channel> 
  <Computer>BenRiach</Computer> 
  <Security /> 
  </System>
- <EventData>
  <Data>TOTALCMD64.EXE</Data> 
  <Data>8.0.1.0</Data> 
  <Data>00000000</Data> 
  <Data>USER32.dll</Data> 
  <Data>6.1.7601.17514</Data> 
  <Data>4ce7c9f1</Data> 
  <Data>c0000005</Data> 
  <Data>00000000000125f8</Data> 
  <Data>c64</Data> 
  <Data>01cdb36b4bbd53a0</Data> 
  <Data>C:\Program Files (x86)\totalcmd\TOTALCMD64.EXE</Data> 
  <Data>C:\Windows\system32\USER32.dll</Data> 
  <Data>69e52259-1f68-11e2-ad4d-005056c00008</Data> 
  </EventData>
  </Event>

[ghisler(Author)]

Unfortunately there is a bug in the Windows Vista/7 delete function: It deletes more files than told when sending files from multiple directories to it. Therefore the sync tool has to delete each directory separately. If there are many subdirectories in sync, you will get the same problem with many parallel delete threads. Unfortunately there is no solution to this Windows bug, sorry.