Synchronize directories with network drive - memory usage

[tcOne]

Hello all,

I'm experiencing the following issue, which I've managed to successfully reproduce at will:

I’m using the Synchronize directories function, having a local directory on the left panel (not very deep path, only english characters, no spaces) and a network directory on the right panel (similarly not very deep path, only english characters, no spaces, in \\192.168.x.x\).

The directories contain 2.800 files each (I’m using the dot as the thousands separator), totaling 20 Mb. Almost all of the files are EPS, around 3-4 Kb in size.

Options used: All checkboxes are checked. I’ve toggled the "Use plugin compare functions" but it made no difference. It is simply set to compare text files (some extensions) as text (tc.Compare as text).

Immediately after pressing the Compare button, the memory usage climbs to the top. If I leave it for a few seconds, the paging file gets to work, slowing everything to a crawl. Afterwards, in this case, it took a good 7 minutes or so before I managed to press the Abort button and another couple more for things to return to close to normal (the disk is still churning away like there’s no tomorrow). Eventually, memory usage returns to normal (after, of course, having caused everything to become paged out).

Now, the strange thing is that the allocated memory is not displayed as being in use by any program (I’m using Process Hacker for task manager). It’s just used by who-knows-what.

This only happens when any one panel (or both) is a network file repository. It does not happen when both panels (i.e. directories to compare) are on local or removable (such as USB) drives. Assigning a drive letter to the network path also made no difference.

Environment: Total Commander 8.52a x86 on Windows 7 Professional SP1 x64, 8Gb RAM, all updates installed (except Windows 10 and telemetry stuff). The network device is a D-Link DNS-345 with the latest firmware I managed to install.

I’ve tested with a fresh wincmd.ini (i.e. closed TC, and deleted the ini) and the result is the same.

I’ve also tried disabling the antivirus, to no avail.

The solution I resorted to was to simply select about 600 files for each comparison step (with the Only selected (in main window) option checked).

I can provide additional information or try any suggestions to solve or troubleshoot this issue.


Kind regards

[gdpr deleted 6]

First, try TC 64bit and see if the problem still occurs.

If the problem still does exist when using TC 64bit, then test with another tool that synchronizes directories (for example http://www.freefilesync.org [EDIT: Don't, the FreeFileSync installer contains OpenCandy ad ware], but there are many more such tools out there) to see whether the problem only occurs when using TC or whether it also occurs if you use other software doing similar things.

I would not be surprised if the problem you witness has nothing to do with TC at all, considering that you seem to have confirmed that it is not TC which is allocating memory excessively.

If the memory really does not belong to a "normal" process (check again, perhaps you made a mistake), then it might be that some system/kernel mode component has allocated that memory (virus scanners, firewalls and other software monitoring/analyzing network traffic come to my mind; although you said you disabled your AV, hmm...)

(Side note: From time to time, I do directory synchronization involving shared network folders, sometimes involving folders with several 10000 files. I never encountered a problem as you describe. Win 7 Pro x64 SP1 and TC 64bit here...)

[Dalai]

I’ve tested with a fresh wincmd.ini (i.e. closed TC, and deleted the ini) and the result is the same.

Well, then it's no TC issue at all, let alone a bug in TC (OK, a bug is still possible).

Unfortunately, I don't have any idea what could cause such an issue... But I concur with elgonzo: It's probably some software deeper in the OS, some driver perhaps; maybe network driver since the issue occurs with network shares/drives.

Regards
Dalai

[tcOne]

Hello elgonzo and Dalai,


I've just tried with TC x64 (with my normal ini and with an empty one) and the same issue happens.

I've also tried with WinMerge and, unfortunately, no problem occurs! Under normal circumstances, that would be a nice thing, but as it happens now this confuses me further.

I wanted to try freefilesync too, as I've used it some time ago and found it quite a nice tool, but the installer has had crapware for some time now, so I've declined.

I honestly *feel* it is not an issue caused by TC.

Right here, before continuing, I took a long break in order to test on another computer (also Win 7 Pro x64 SP1, with TC x86). The same issue there too…

HOWEVER

Something strange I’ve noticed...

On my computer, TC performs directory comparisons *fast*, whereas WinMerge was noticeably slower. That is (according to the progress display of the programs), TC compares many more items per second (about 4x) than WinMerge. This is comparison by content in both cases.

The other computer has had the exact same behavior, except that it has a slow network card (10Mbps, don’t ask!:-)), so it wasn’t as quick to have its memory allocated.

It almost certainly looks like the issue here lies squarely on the D-Link. It’s as if, say, there’s something wrong with its speed as far as file handles are concerned. Maybe it is slow to answer to Windows, so the file handles stay active longer, taking up memory (does this make any sense?). Now, how come the memory usage ends up being 4-5Gb for 40Mb of data to compare, it’s anyone’s guess.

Anyway, just to make things clear, it is NOT an issue of TC, so there’s nothing to solve! :-)

In case I do find a solution, I'll add information to this post. Perhaps someone will find it useful.

Regards

[gdpr deleted 6]

Oops, my apologies for pointing you towards freefilesync. I did use it occasionally quite some years ago, and at that time (perhaps 9..10 years ago) it was not bundled with crapware. Again, my apologies for suggesting this crap

I would not suspect a network driver itself to hog that much memory. Even if it would keep thousands of file handles open this would only amount to some megabytes, i guess.

I would rather suspect you have some software on your system (perhaps bundle-ware that came with some network hardware?) that kind of tries to optimize/accelerate access to network shares (think caching) and falls flat on its face, exhausting the available RAM in the process...

Anyway, to verify if it is indeed the network driver as you suspect, you could setup a virtual machine and let the guest OS provide a shared network folder (assuming you don't have a another NIC at hand...).

Make sure that the VM is configured to use a private (host-only) network between the host and the guest OS. The important bit here is that the network driver of your physical network interface will not be part of the (virtual) network between your host OS and the guest OS in the VM.

In your host OS you once more do a test with TC, but this time using the shared network folder from the VM. If the test with the VM still shows excessive RAM usage, you know that it is not the driver of the physical network interface but some other software component in your system that causes the problem. If the problem cannot be reproduced, then in all likelihood the driver of the physical network interface (or something in its respective driver stack) has a hand in causing the problem.

[tcOne]

I've just tried with a Win7 VM (no antivirus or firewall installed at all) and the TC sync dirs function works without issues between the host and the guest.
I've also tried to sync dirs between the guest (VM) and the D-Link and, sure enough, the memory usage went through the roof (only in the VM).
D-Link, I'm looking at you with great suspicion... :-)

[gdpr deleted 6]

I've also tried to sync dirs between the guest (VM) and the D-Link and, sure enough, the memory usage went through the roof (only in the VM).

This is utterly bizarre. How could a CIFS/SMB file server (such as your NAS) cause excessive RAM allocation on a client? Not sure what is going on. Almost paranormal...

Perhaps some esoteric errors in the SMB protocol transmissions from the NAS cause this problem. Wanna check your system event log for errors with regard to networking/SMB?

I feel stupid for asking because you probably took care about it, but is the Win7 guest OS a clean virgin installation without any kind of 3rd-party software or drivers installed? (Well, except TC of course...)

[tcOne]

Thank you elgonzo for helping me out with this. I appreciate your responses very much, moreover since I think it's clear at this point that this is not a bug in TC.

The event log is a labyrinth, so I'm currently searching for information on what should I look out for. :-)

I did fire up Process Monitor, just in case it helps me out.
I'm seeing that for every 8 or so entries (for each small file comparison) there are around 20 like the following:

00:35:08,1948524 TOTALCMD.EXE 6820 CreateFile C:\Windows\CSC\v2.0.6\namespace\192.168.1.73 NAME NOT FOUND Desired Access: Read EA, Write EA, Read Attributes, Write Attributes, Delete, Read Control, Synchronize, Disposition: Open, Options: Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a

I have no idea what this means (if it indeed does mean something) as far as the issue at hand is concerned.
By the way, Process Monitor caused a slight delay in the comparison process, so the result was exactly the same as I noticed on the computer with the slow network interface. The memory allocation wasn’t that quick, but had its ups and downs as the process went merrily along.
Have a look at the nice graph
imgur com/PPG9xjL
The left part is just before pressing Compare and the drop off on the right is when the process ended.

Witchcraft indeed...

[Dalai]

You should enable the column "Peak Working Set" in Task Manager to see which process allocates the memory; you should let Task Manager show all processes for this (launches Task Manager as administrator). If you really don't see anything in Task Manager you should check Resouce Monitor.

Regards
Dalai

[tcOne]

Hello Dalai,

I usually have Process Hacker running (with admin rights) at all times. I've just added the Peak Working Set column, but it didn't show anything new. The size reported remained the same before, during and after the comparison (which I stopped before paging starts).

edit:

I've just tried Resource Monitor. On the Memory tab, the In Use number runs to the top after I press Compare, but nothing changes in the grid of process entries above.

[milo1012]

I recommend RamMap for a quick overview of system mem usage.
Maybe do some screenshots of it while the mem usage is high.

[gdpr deleted 6]

00:35:08,1948524 TOTALCMD.EXE 6820 CreateFile C:\Windows\CSC\v2.0.6\namespace\192.168.1.73 NAME NOT FOUND Desired Access: Read EA, Write EA, Read Attributes, Write Attributes, Delete, Read Control, Synchronize, Disposition: Open, Options: Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a

I have no idea what this means (if it indeed does mean something) as far as the issue at hand is concerned.

It should have nothing to do with the problem at hand. The purpose of the CSC directory is to hold offline copies of files and directories from a network share. Any process accessing a file on a network share will access the CSC (more precisely, certain I/O WINAPI functions called by the processes will do this). If you right-click on a file or directory in a network share and select "Always available offline" in the context menu, then a copy of that file or directory will be created in the CSC and kept in sync with the original file/directory on the network share. If a process tries to access a file on a network share while that network share is not available, then the respective I/O operations will be redirected to the offline copy of the file in the CSC.

[tcOne]

Hello milo1012,

here is a screenshot of RamMap.
imgur com/cINlOYL
As I understand, it "runs" the moment it starts and whenever I press refresh. This is after I refreshed, when the memory usage hit the roof.

Here is a screenshot from Process Hacker's System Information Memory tab. I took it a few seconds after the event.
imgur com/8BIDMK8

I still have RamMap active (haven't refreshed it since the screenshot above) but I see no process with excessive numbers.

[milo1012]

here is a screenshot of RamMap.
imgur com/cINlOYL
As I understand, it "runs" the moment it starts and whenever I press refresh. This is after I refreshed, when the memory usage hit the roof.

Yes, the large Paged Pool basically confirms that it is either a driver or a network stack issue, i.e. Windows reserves an extreme amount of memory for kernel mode operations for some reason.
That way it will not be easy to identify the exact issue, you just know that it's an Windows internal flaw for now.

Paged pool, on the other hand, gets its name from the fact that Windows can write the data it stores to the paging file, allowing the physical memory it occupies to be repurposed. Just as for user-mode virtual memory, when a driver or the system references paged pool memory that’s in the paging file, an operation called a page fault occurs, and the memory manager reads the data back into physical memory. The largest consumer of paged pool, at least on Windows Vista and later, is typically the Registry, since references to registry keys and other registry data structures are stored in paged pool. The data structures that represent memory mapped files, called sections internally, are also stored in paged pool.

(So it might also be memory mapped files, but this shouldn't be the case when the files are reasonably small)

[tcOne]

I see...

thank you all for the assistance in troubleshooting this.

I'm now fairly convinced that the culprit is D-Link's network stack implementation. I feel it is not productive to spend any more time in this issue, seeing as the device is not exactly current (it is about 4 years old).

Thank you all again for all the advice.

Regards