TC freezes inside folder with hundreds of thousands of files, but Explorer works ok... :shock:

English support forum

Moderators: white, Hacker, petermad, Stefan2

User avatar
Hacker
Moderator
Moderator
Posts: 13052
Joined: 2003-02-06, 14:56 UTC
Location: Bratislava, Slovakia

Re: TC freezes inside folder with hundreds of thousands of files, but Explorer works ok... :shock:

Post by *Hacker »

Christian,
Well, the point here is that in Explorer such a delay is not observed.

Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.
User avatar
jesped
Member
Member
Posts: 149
Joined: 2005-02-07, 03:28 UTC

Re: TC freezes inside folder with hundreds of thousands of files, but Explorer works ok... :shock:

Post by *jesped »

ghisler(Author) wrote: 2021-10-26, 16:13 UTC What hardware do you have? I'm using an old core i5 first generation as my main development PC, and even with that it takes 5 seconds to enter a folder with 130000 subfolders, and 7 seconds to switch from full to brief view. However, I'm using an SSD now, which makes this much faster than with a harddisk.
In my case there are 213462 files, with a total size of 1,239.3 G

Tried on 2 machines, both with Ryzen9 3900X 4.0Ghz. Disk is a Toshiba DT01ACA300 3TB.

But it doesn't have anything to do with hardware, because it works perfectly fluid in both machines with Windows Explorer.
You can see in the captured videos just how evident and huge the difference is in responsiveness both entering and navigating the same folder.

As said, unless this is a TC core problem somehow, I really don't know why there is such a big difference that makes TC behave so sluggish and unresponsive compared to Windows Explorer.
Troy_Daniel
Junior Member
Junior Member
Posts: 14
Joined: 2021-04-30, 14:14 UTC

Re: TC freezes inside folder with hundreds of thousands of files, but Explorer works ok... :shock:

Post by *Troy_Daniel »

jesped wrote: 2021-10-27, 14:41 UTC
ghisler(Author) wrote: 2021-10-26, 16:13 UTC What hardware do you have? I'm using an old core i5 first generation as my main development PC, and even with that it takes 5 seconds to enter a folder with 130000 subfolders, and 7 seconds to switch from full to brief view. However, I'm using an SSD now, which makes this much faster than with a harddisk.
In my case there are 213462 files, with a total size of 1,239.3 G

Tried on 2 machines, both with Ryzen9 3900X 4.0Ghz. Disk is a Toshiba DT01ACA300 3TB.

But it doesn't have anything to do with hardware, because it works perfectly fluid in both machines with Windows Explorer.
You can see in the captured videos just how evident and huge the difference is in responsiveness both entering and navigating the same folder.

As said, unless this is a TC core problem somehow, I really don't know why there is such a big difference that makes TC behave so sluggish and unresponsive compared to Windows Explorer.
This situation happens to me. Once I dealt with a large quantity html files, abaut 200k, TC takes a long period of time to enter the folder, while for the explorer, it enters quite soon. (My list doesn't show any icons and in the default list view.)

I think what make the difference is that, Expolorer loads only part of the file list, just like lazy load mode for a webpage, while for TC, it loads the whole file list (possibly to get the write status information).

Finally, my solution is to use the commandline (cmd or bash in msys2) to access files in the folder.
User avatar
jesped
Member
Member
Posts: 149
Joined: 2005-02-07, 03:28 UTC

Re: TC freezes inside folder with hundreds of thousands of files, but Explorer works ok... :shock:

Post by *jesped »

If this is a problem of how TC core handles reading folders compared to Explorer, then definitely there should be an option to read them the same way Explorer does, because it's night and day how responsive Explorer is vs how sluggish TC is.
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: TC freezes inside folder with hundreds of thousands of files, but Explorer works ok... :shock:

Post by *ghisler(Author) »

Explorer loads only part of the file list
Yes it does - and when you check, it takes longer until it finishes loading everything than Total Commander. And until it does, the sort order is of course not correct.
Author of Total Commander
https://www.ghisler.com
User avatar
jesped
Member
Member
Posts: 149
Joined: 2005-02-07, 03:28 UTC

Re: TC freezes inside folder with hundreds of thousands of files, but Explorer works ok... :shock:

Post by *jesped »

ghisler(Author) wrote: 2021-10-31, 09:15 UTC
Explorer loads only part of the file list
Yes it does - and when you check, it takes longer until it finishes loading everything than Total Commander. And until it does, the sort order is of course not correct.
Is it possible for TC to have an option to read them like Explorer does so it doesn't freeze/stall?

I hate to insist on this, but it is literally impossible to manage these kind of folders in TC like this. I had to resort to open Explorer every time I have to do something with these folders, which is becoming more and more frequently.
User avatar
petermad
Power Member
Power Member
Posts: 14741
Joined: 2003-02-05, 20:24 UTC
Location: Denmark
Contact:

Re: TC freezes inside folder with hundreds of thousands of files, but Explorer works ok... :shock:

Post by *petermad »

A suggestion:

Since you need NTFS (or exFAT) partitions to hold more than 65,534 files ion a folder and since NTFS is sorted by name I suggest that if TC is set to unsorted View (Crtl+F7) then only read a few 100 files ahead.

BTW just using unsorted view already make a diference:
I made a folder with 200000 test files on an NTFS volume. If TC is set to name sorting it takes about 9 seconds to enter the folder - if I set TC to unsorted view it takes about 3 seconds to load the directory.

jesped have you tried shifting to unsorted view in TC before entering one of those giant folders - any difference for you?
Last edited by petermad on 2021-11-02, 22:19 UTC, edited 1 time in total.
License #524 (1994)
Danish Total Commander Translator
TC 11.03 32+64bit on Win XP 32bit & Win 7, 8.1 & 10 (22H2) 64bit, 'Everything' 1.5.0.1371a
TC 3.50b4 on Android 6 & 13
Try: TC Extended Menus | TC Languagebar | TC Dark Help | PHSM-Calendar
User avatar
jesped
Member
Member
Posts: 149
Joined: 2005-02-07, 03:28 UTC

Re: TC freezes inside folder with hundreds of thousands of files, but Explorer works ok... :shock:

Post by *jesped »

petermad wrote: 2021-11-02, 18:00 UTC jesped have you tried shifting to unsorted view in TC before entering one of those giant folders - any difference for you?
Well... I need them sorted by date or filename. Explorer doesn't have any problem with that. I can enter the folder, switch the sorting to anything and even while generating thumbnails and even while writing new files to folder... meanwhile TC is totally unusable.

You can clearly see it in the videos. This is a real case, not a test with some empty files. 213462 files, a total size of 1,239.3 G:

https://youtu.be/9A8O5OyzveI

11 seconds just to enter the folder (00:00:07 to 00:00:18), and then constant unresponsiveness.

Changing from Full view mode to Brief... one minute and 3 seconds ( 00:00:23 to 00:01:26 ). :shock:

Meanwhile in Windows Explorer... https://youtu.be/0OIfnVK0fWc

I don't know how anybody can be saying that TC is doing things right/better and Explorer wrong/worse... or why TC can't do the same as Explorer does.
User avatar
Dalai
Power Member
Power Member
Posts: 9365
Joined: 2005-01-28, 22:17 UTC
Location: Meiningen (Südthüringen)

Re: TC freezes inside folder with hundreds of thousands of files, but Explorer works ok... :shock:

Post by *Dalai »

Although this is OT, I need to jump in on this.
petermad wrote: 2021-11-02, 18:00 UTCSince you need NTFS (or exFAT) partitions to hold more than 65,534 files
This is worded a little bit too vague. Even FAT32 volumes/partitions can hold much more than 2^16 files, just not in a single directory. The maximum number of files per directory also depends on the length of the filenames. See https://superuser.com/questions/446282/max-files-per-directory-on-ntfs-vol-vs-fat32 and https://blog.paddlefish.net/?page_id=1017 for reference.

Regards
Dalai
#101164 Personal licence
Ryzen 5 2600, 16 GiB RAM, ASUS Prime X370-A, Win7 x64

Plugins: Services2, Startups, CertificateInfo, SignatureInfo, LineBreakInfo - Download-Mirror
User avatar
petermad
Power Member
Power Member
Posts: 14741
Joined: 2003-02-05, 20:24 UTC
Location: Denmark
Contact:

Re: TC freezes inside folder with hundreds of thousands of files, but Explorer works ok... :shock:

Post by *petermad »

Dalai wrote:This is worded a little bit too vague
I ment files pr. folder - sorry I have now changed that in the post.

jesped wrote:Well... I need them sorted by date or filename.
OK - then of course my suggestion is not an option.
License #524 (1994)
Danish Total Commander Translator
TC 11.03 32+64bit on Win XP 32bit & Win 7, 8.1 & 10 (22H2) 64bit, 'Everything' 1.5.0.1371a
TC 3.50b4 on Android 6 & 13
Try: TC Extended Menus | TC Languagebar | TC Dark Help | PHSM-Calendar
User avatar
petermad
Power Member
Power Member
Posts: 14741
Joined: 2003-02-05, 20:24 UTC
Location: Denmark
Contact:

Re: TC freezes inside folder with hundreds of thousands of files, but Explorer works ok... :shock:

Post by *petermad »

2jesped
Meanwhile in Windows Explorer... https://youtu.be/0OIfnVK0fWc
Your video does't show changing the sorting - that could be interesting - not how fast, I am sure Explorer is faster, but what columns are updated first - i noticed that the Date column was lagging when scrolling in Explorer in Details view.
License #524 (1994)
Danish Total Commander Translator
TC 11.03 32+64bit on Win XP 32bit & Win 7, 8.1 & 10 (22H2) 64bit, 'Everything' 1.5.0.1371a
TC 3.50b4 on Android 6 & 13
Try: TC Extended Menus | TC Languagebar | TC Dark Help | PHSM-Calendar
User avatar
nsp
Power Member
Power Member
Posts: 1804
Joined: 2005-12-04, 08:39 UTC
Location: Lyon (FRANCE)
Contact:

Re: TC freezes inside folder with hundreds of thousands of files, but Explorer works ok... :shock:

Post by *nsp »

Hi Jesped,

Do you run TC as normal user, 32bit or 64bit version ?
Did you disabled thumbnail cache ?
Can you try without any icon (Configuration/Display/Icons)
Have you made a try with an empty wincmd.ini and none of the Everything option selected. (tree,size,...)

Have you a rule in your antivirus that exclude TC from detection and the same for external drive ?
Is it the same behavior if you first navigate with Explorer and then use TC ?
User avatar
jesped
Member
Member
Posts: 149
Joined: 2005-02-07, 03:28 UTC

Re: TC freezes inside folder with hundreds of thousands of files, but Explorer works ok... :shock:

Post by *jesped »

nsp wrote: 2021-11-03, 07:35 UTC Do you run TC as normal user, 32bit or 64bit version ?
TC Version 10 x64. My Windows user is an Admin, but TC is run in normal mode.
nsp wrote: 2021-11-03, 07:35 UTCDid you disabled thumbnail cache ?
Yes
nsp wrote: 2021-11-03, 07:35 UTCCan you try without any icon (Configuration/Display/Icons)
As stated in this thread, already did and didn't change anything.
nsp wrote: 2021-11-03, 07:35 UTCHave you made a try with an empty wincmd.ini and none of the Everything option selected. (tree,size,...)
As stated in this thread, already did and didn't change anything.
nsp wrote: 2021-11-03, 07:35 UTCHave you a rule in your antivirus that exclude TC from detection and the same for external drive ?
Antivirus is disabled.
nsp wrote: 2021-11-03, 07:35 UTCIs it the same behavior if you first navigate with Explorer and then use TC ?
Yes.

The problem here is Explorer does what it needs to do in the background, and shows you what is available as fast as it can, while TC tries to read everything and meanwhile blocks/freezes you out of any interaction and then still behaves very sluggish and unresponsive.

Explorer shows you the filenames and allows you to do a smooth navigation while it reads the rest of the details. If you move to a different place in the list then it jumps to start reading what is there, what you are seeing on your screen. It's always smooth and allows you to go forward, back, change the view type, sort order, whatever, ... and never blocks/freezes your interaction. At worst you just have to wait a bit for it to finish in the background, BUT you can smoothly do whatever you want meanwhile, unlike in TC...
User avatar
mtech
New Member
New Member
Posts: 1
Joined: 2011-05-09, 17:33 UTC

Re: TC freezes inside folder with hundreds of thousands of files, but Explorer works ok... :shock:

Post by *mtech »

I am also having the same problem with small files on an 8Tb Seagate IronWolf Pro
Main Folder is 841Gb with 633000 files in 34184 folders

Midi sub-directory 13.7 Gb 308220 Files 27000 folders (.mid file sizes are 15k - 100k max. )
MP3 sub-Directory 575 Gb in 158000 Files in 7081 Folders (.mp3 files are 3 - 5 Mb each Max )
OTHER Midi Sub-Directory 198 Gb in 157000 Files in 7000 Folders (.mid file sizes are 15k - 100k max. )

"My Music" is also on the 8Tb Drive 763 Gb 83243 Files in 8356 folders But doesn't have any delays
calling up Music Tracks in .mp3 or .flac


My Machine Specs are as follows:
Windows 10 x 64 Pro - Intel 3770K i7 CPU - Intel DH77KC Board - 32Gb DDR3-1600Mhz RAM - Boot 1 Tb M2 SATA drive, Storage 8Tb Seagate IronWolf Pro
Post Reply