Huge number of files

[devzero]

Totalcommander is ekstremly slow when working with a huge number of files. Typicaly i have a directory with 30000+ files inn it, and it takes from 2-5 minutes to enter it.

Is there any workaround to this? Any other software i can use to work with this amount of files?

[gbo]

This is probably not what you are looking for, but this plugin allow to separate files into folders. It would improve performance but I guess your files have to be in the same folder

[Hacker]

devzero,

Totalcommander is ekstremly slow when working with a huge number of files. Typicaly i have a directory with 30000+ files inn it, and it takes from 2-5 minutes to enter it.

Try Configuration - Options - Display - Show symbols to the left of the filename - No symbols.

Any other software i can use to work with this amount of files?

Well, command line should be fast.

HTH
Roman

[devzero]

Ill test the plugin later today, if it can keep the speed up it would be a lifesaver.

I should probably have mentioned that commandline is mostly to slow also, i had a directory that i ran
dir /b > ../dir.txt
After running for 15 minnutes the txt file was 16mb and still running.

Doing a
del *.* /Q took about 10 minutes.

And no, i have no idea exactly how many files that directory contained.

[Hacker]

Maybe it would be better not to have so many files in a dir?

Roman

[devzero]

That is unfortanatly not an option all the time for me

[Lefteous]

2devzero
two questions:

1) Is directory opening slow the first time only or even in repeated openings?
Please report the cpu usage while the directory is read.

2) What is the performance in Windows Explorer?

[devzero]

1) It is slow everytime, if i could just wait while it opened the first time, then be able to work with the files normaly it would be ok. Problem is not only is it slow on first open, but somtimes it seems to "autorefresh" when i select it and takes another several minutes to do this.

2) Explorer is actually much worse.

This was tested on windows 2000 adv. server with ntfs file system and a fairly fast SCSI disk, altho i don't know exact spec's on the disk. The dir contains 38000 files and 30000 dirs.

[Lefteous]

2devzero

It is slow everytime, if i could just wait while it opened the first time, then be able to work with the files normaly it would be ok. Problem is not only is it slow on first open, but somtimes it seems to "autorefresh" when i select it and takes another several minutes to do this.

OK to disable all possible types of refrehes do the following:
In main menu call Configuration/Change Settings Files directly.
In Wincmd.ini changed/add the following settinga:

Code: Select all

NoReRead=ABCDEFGHIJKLMNOPQRSTUVWXYZ\
WatchDirs=0

Restart Total Commander after applying this settings.

Please add the cpu usage information.

Explorer is actually much worse.

After all it seems Total Commander is the fastest program of all.

The dir contains 38000 files and 30000 dirs.

Ok that means we are not talking about 30000 but about 68000 objects in that particular directory.

Another idea:
Maybe you could use a filter like *.jpg if don't need to see all files at once.

[CoolWater]

May be OT:

At this point, the speed of SpeedCommander could be quite interessting because (as far as I know) SC does not read all entries but only the amount of items which would be visible without scrolling. As soon you scroll the list, both names and icons are read. All non-visible items are empty, no icon, no text... (see: http://blog.speedproject.de/2005/10/26/verkuerzte-baumansicht/ (german blog entry))

TC does it different: Read all entries (all names), extract icons (if not disabled) only for those which are visible.

Correct me if i'm wrong...

Regards,
CoolWater

[devzero]

CPU usage is from 10 to 20% on total commander, other prosesses uses next to nothing. (when totcmd used 20%, total system load was about 22%) for the first few minnutes, last 30 sek totcmd used 50% (or 100% of one CPU as this system has 2 cpu's )

Now off to test speedcommander...

[aguirRe]

I don't know if it helps here, but I know there's an NTFS tweak that is supposed to help when having massive amounts of files/dirs. It's a registry key called NtfsDisableLastAccessUpdate (search Google for more info).

[CoolWater]

CPU usage is from 10 to 20% on total commander, other prosesses uses next to nothing. (when totcmd used 20%, total system load was about 22%) for the first few minnutes, last 30 sek totcmd used 50% (or 100% of one CPU as this system has 2 cpu's )

Now off to test speedcommander...

may be you should turn off your anti virus program for testing...

[Lefteous]

2devzero

CPU usage is from 10 to 20% on total commander, other prosesses uses next to nothing. (when totcmd used 20%, total system load was about 22%) for the first few minnutes, last 30 sek totcmd used 50% (or 100% of one CPU as this system has 2 cpu's

This is a very interesting information. This sounds as if "the first minutes" are used to get the files and last 30 seconds to display them.

[devzero]

No antivirus program running on this computer.