Improve handling folders with HUGE number of files

Here you can propose new features, make suggestions etc.

Moderators: white, Hacker, petermad, Stefan2

Post Reply
mariush
Junior Member
Junior Member
Posts: 4
Joined: 2018-09-25, 20:40 UTC

Improve handling folders with HUGE number of files

Post by *mariush »

I realize this is sort of an extreme case, so I don't mind if it would be considered extremely low priority or not worth investigating, but I figured it doesn't hurt to ask.

From time to time, I have to mirror some websites (legally) or update offline copies of websites, and due to the structure of those websites I end up with folders that have huge amount of files in some folders.
For example, right now one website I'm dealing with has over 300'000 files in a single folder (the website uses url rewriting schemes to make the URL's pretty).

- a bit offtopic but - On Windows, Microsoft has implemented NTFS in such a way that when you have over around 10k files in a folder, everything becomes a bit sluggish (access times to files, opening and closing files, everything has some latency). The more files, the higher the latencies involved.
With the mirroring software constantly accessing downloaded files to rewrite urls inside the downloaded pages for offline browsing, you can imagine even with a SSD there would be super high access times and disk load.
I'm getting around that when mirroring websites by using a 8 GB ram drive, basically holding all the files in RAM instead of using disks and configured the ram disk software to make snapshots (disk images) of the ram drive every 30 minutes.

Anyway, back on topic. With TC 9.12 that I had installed, once the number of files got above around 200k, TC started to randomly refuse to read the contents of the folder.
I don't know how it would behave with an actual hdd or ssd (i imagine it would be way slower) but even with the ram drive, when double clicking on a folder TC spends about 5s to acknowledge and then the "reading folder" shows up and then it takes maybe 10-20s to read the contents of the folder. With this older version of TC, maybe 4 out of 5 times, TC would fail to list the contents with just a beep once the file count got over around 250k files. Opening with Windows Explorer works, all files are shown, and I get an initial bunch of files and then I see in the bottom of the explorer windows as it keeps reading chunks of files until all 250k+ are read.
Adding to this, every time TC loses focus and then I set focus back on TC (for example, i drag a html file from that folder to the web browser on the second monitor to check if the software saved the html file correctly), TC tries to refresh the contents of the folder and sometimes fails.

Before writing this message, i downloaded the latest version 9.21 to see if something has improved and after around 3-5 tries, I see it works listing 305k+ files but is still really slow.

I imagine the internals are probably optimized to be really fast and use less memory but I'm wondering if the reason for taking so long time to read the contents of such big folders is because the software is allocating small buffers or has to constantly resize buffers and do memory copies.

I'm wondering if it would be possible to have some "internal" option or something that would say "it's ok to use more memory to improve speed" or "remember that this folder had a lot of files last time you checked it, so it's ok to allocate a bigger buffer right away to speed file list generation"
I'd be fine with hiding such option in the ini file, or in the options > advanced entry or something

-- Thank you for taking time to read this
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Improve handling folders with HUGE number of files

Post by *ghisler(Author) »

Are you using TC 32-bit or 64-bit? The 64-bit version shouldn't run out of memory when reading such huge folders, but the 32-bit version may.
Author of Total Commander
https://www.ghisler.com
User avatar
Usher
Power Member
Power Member
Posts: 1675
Joined: 2011-03-11, 10:11 UTC

Re: Improve handling folders with HUGE number of files

Post by *Usher »

2mariush
Have you tried TC with icons turned off?
Andrzej P. Wozniak
Polish subforum moderator
Post Reply