Slow directory reading

English support forum

Moderators: Stefan2, white, sheep, Hacker

User avatar
Hacker
Moderator
Moderator
Posts: 11213
Joined: 2003-02-06, 14:56 UTC
Location: Bratislava, Slovakia

Slow directory reading

Post by *Hacker » 2019-03-20, 10:20 UTC

Hi,
TC 9.22 x64 on Windows 7 x64. Old laptop with a 5400 RPM 7200 RPM hard drive (not sleeping).

I have an inactive tab which points to a directory with ~2500 files and is sorted by date. I have two color filters, one for NTFS compressed files and one for files which contain any descript.ion comment (tc comment regex .). None of the files match the color filter criteria, if it's relevant in any way.

Now when I click the tab, it takes about 5 seconds 20 seconds to show the files. For comparison, Everything shows all ~700 thousand files on my hard drive sorted by date instantaneously (no color filters of course).
Is there any way to make reading and showing the contents of this tab faster (save for buying a faster hard drive / SSD)?

TIA
Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 37363
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Slow directory reading

Post by *ghisler(Author) » 2019-03-20, 11:20 UTC

Try disk defragmentation via right click on the drive letter in TC - Properties.
Author of Total Commander
http://www.ghisler.com

User avatar
Hacker
Moderator
Moderator
Posts: 11213
Joined: 2003-02-06, 14:56 UTC
Location: Bratislava, Slovakia

Re: Slow directory reading

Post by *Hacker » 2019-03-20, 11:40 UTC

Christian,
Thank you. The analysis says the disk is 0% fragmented but I will give it a try anyways. Until then, any other ideas I might try as well?

TIA
Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.

User avatar
Usher
Senior Member
Senior Member
Posts: 376
Joined: 2011-03-11, 10:11 UTC

Re: Slow directory reading

Post by *Usher » 2019-03-20, 12:36 UTC

Turn off icons and compare results.
Regards from Poland
Andrzej P. Wozniak

User avatar
Hacker
Moderator
Moderator
Posts: 11213
Joined: 2003-02-06, 14:56 UTC
Location: Bratislava, Slovakia

Re: Slow directory reading

Post by *Hacker » 2019-03-21, 02:35 UTC

Usher,
Hmm, good idea, but all files are *.txt files so that shouldn't be a problem. I'll try anyways.

Also it seems I have underestimated the read time by quite a lot in my original post, since after defragmentation it is ~20 seconds for a directory of ~2500 *.txt files (sorted by date, no matching color filters) and ~5 seconds for a directory of ~700 *.txt files (same criteria as the previous dir).

Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.

User avatar
Dalai
Power Member
Power Member
Posts: 6568
Joined: 2005-01-28, 22:17 UTC
Location: Meiningen (Südthüringen)

Re: Slow directory reading

Post by *Dalai » 2019-03-21, 03:56 UTC

Is it possible that the disk has trouble reading sectors or is even dying? Take a look at the SMART values with CrystalDiskInfo and maybe check the (theoretical) performance with HD Tune.

Regards
Dalai
#101164 Personal licence
Athlon X4 880K, 16 GiB RAM, Gigabyte F2A88X-D3HP, Win7 x64

Plugins: Services2, Startups

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 37363
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Slow directory reading

Post by *ghisler(Author) » 2019-03-21, 09:23 UTC

These are extremely slow read times even for a hdd!

It's possible that defrag didn't do anything to the directory because the fragmentation level was too small.
Here is a small tool I wrote in 2006 which can show the number of fragments of a file or directory, and their location on disk:
https://www.totalcommander.ch/win/showfrag.zip

I have signed it with my certificates now to show that it's authentic.
Author of Total Commander
http://www.ghisler.com

User avatar
makinero
Senior Member
Senior Member
Posts: 268
Joined: 2013-10-26, 10:05 UTC

Re: Slow directory reading

Post by *makinero » 2019-03-21, 10:25 UTC

I have already written about it many times. TC is not adapted to read large directories, and in particular with small files, where speed drops drastically.
I know from experience that people try various ways to improve speed a) defragmentation b) check SMART
But is real? It will not help, the speed will not improve. Although you can try, and the results are always deplorable.
Disable showing icons in TC, should in some ways help, but all in all: There are tools whose icons do not affect speed.

User avatar
Hacker
Moderator
Moderator
Posts: 11213
Joined: 2003-02-06, 14:56 UTC
Location: Bratislava, Slovakia

Re: Slow directory reading

Post by *Hacker » 2019-03-21, 11:14 UTC

Christian,
Thank you for showfrag. For both the 20 seconds dir with ~2500 files and the 5 seconds dir with ~700 files it shows "Number of fragments: 1".

makinero,
I do not really consider a dir with ~2500 files especially large.

Dalai,
CrystalDiskInfo shows the health as "Good":

Code: Select all

----------------------------------------------------------------------------
CrystalDiskInfo 8.0.0 (C) 2008-2018 hiyohiyo
                                Crystal Dew World : https://crystalmark.info/
----------------------------------------------------------------------------

    OS : Windows 7 Enterprise SP1 [6.1 Build 7601] (x64)
  Date : 2019/03/21 11:40:27

----------------------------------------------------------------------------
 (1) TOSHIBA MK5061GSYN
----------------------------------------------------------------------------
           Model : TOSHIBA MK5061GSYN
        Firmware : MH000C
       Disk Size : 500,1 GB (8,4/137,4/500,1/500,1)
     Buffer Size : 16384 KB
     Queue Depth : 32
    # of Sectors : 976773168
   Rotation Rate : 7200 RPM
       Interface : Serial ATA
   Major Version : ATA8-ACS
   Minor Version : ----
   Transfer Mode : SATA/300 | SATA/300
  Power On Hours : 11736 hours
  Power On Count : 1525 count
     Temperature : 53 C (127 F)
   Health Status : Good
        Features : S.M.A.R.T., APM, 48bit LBA, NCQ
       APM Level : 0080h [ON]
       AAM Level : ----
    Drive Letter : C:

-- S.M.A.R.T. --------------------------------------------------------------
ID Cur Wor Thr RawValues(6) Attribute Name
01 100 100 _50 000000000000 Read Error Rate
02 100 100 _50 000000000000 Throughput Performance
03 100 100 __2 0000000008DF Spin-Up Time
04 100 100 __0 00000000061E Start/Stop Count
05 100 100 _10 000000000000 Reallocated Sectors Count
07 100 100 _50 000000000000 Seek Error Rate
08 100 100 _50 000000000000 Seek Time Performance
09 _71 _71 __0 000000002DD8 Power-On Hours
0A 131 100 _30 000000000000 Spin Retry Count
0C 100 100 __0 0000000005F5 Power Cycle Count
B7 100 100 __1 000000000003 Vendor Specific
B8 100 100 _97 000000000000 End-to-End Error
B9 100 100 __1 00000000FFFF Vendor Specific
BB 100 100 __0 000000000000 Reported Uncorrectable Errors
BC 100 _99 __0 000000000028 Command Timeout
BD 100 100 __1 000000000000 High Fly Writes
BE _47 _36 _40 002035150035 Airflow Temperature
BF 100 100 __0 000000000012 G-Sense Error Rate
C0 100 100 __0 000000350035 Power-off Retract Count
C1 _96 _96 __0 00000000A6ED Load/Unload Cycle Count
C4 100 100 __0 000000000000 Reallocation Event Count
C5 100 100 __0 000000000000 Current Pending Sector Count
C7 200 200 __0 000000000000 UltraDMA CRC Error Count
GSMartControl also shows "Passed":

Code: Select all

ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   100   100   050    -    0
  2 Throughput_Performance  POS--K   100   100   050    -    0
  3 Spin_Up_Time            PO---K   100   100   002    -    2271
  4 Start_Stop_Count        -O--CK   100   100   000    -    1566
  5 Reallocated_Sector_Ct   PO--CK   100   100   010    -    0
  7 Seek_Error_Rate         POSR-K   100   100   050    -    0
  8 Seek_Time_Performance   P-S--K   100   100   050    -    0
  9 Power_On_Minutes        -O--CK   071   071   000    -    195h+37m
 10 Spin_Retry_Count        PO--CK   131   100   030    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    1525
183 Runtime_Bad_Block       -O--CK   100   100   001    -    3
184 End-to-End_Error        PO--CK   100   100   097    -    0
185 Unknown_Attribute       -O--CK   100   100   001    -    65535
187 Reported_Uncorrect      -O--CK   100   100   000    -    0
188 Command_Timeout         -O--CK   100   099   000    -    40
189 High_Fly_Writes         -O-RCK   100   100   001    -    0
190 Airflow_Temperature_Cel -O---K   048   036   040    Past 52 (Min/Max 21/53 #32)
191 G-Sense_Error_Rate      -O--CK   100   100   000    -    18
192 Power-Off_Retract_Count -O---K   100   100   000    -    3473461
193 Load_Cycle_Count        -O--CK   096   096   000    -    42733
196 Reallocated_Event_Count -O--CK   100   100   000    -    0
197 Current_Pending_Sector  -O--CK   100   100   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning
HD Tune looks more or less OK, I guess?

However, CrystalDiskMark results look abysmal:

Code: Select all

-----------------------------------------------------------------------
CrystalDiskMark 6.0.2 x64 (C) 2007-2018 hiyohiyo
                          Crystal Dew World : https://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

   Sequential Read (Q= 32,T= 1) :    57.773 MB/s
  Sequential Write (Q= 32,T= 1) :    58.303 MB/s
  Random Read 4KiB (Q=  8,T= 8) :     0.315 MB/s [     76.9 IOPS]
 Random Write 4KiB (Q=  8,T= 8) :     0.684 MB/s [    167.0 IOPS]
  Random Read 4KiB (Q= 32,T= 1) :     0.323 MB/s [     78.9 IOPS]
 Random Write 4KiB (Q= 32,T= 1) :     0.737 MB/s [    179.9 IOPS]
  Random Read 4KiB (Q=  1,T= 1) :     0.298 MB/s [     72.8 IOPS]
 Random Write 4KiB (Q=  1,T= 1) :     0.811 MB/s [    198.0 IOPS]

  Test : 1024 MiB [C: 96.6% (449.9/465.7 GiB)] (x5)  [Interval=5 sec]
  Date : 2019/03/21 11:38:18
    OS : Windows 7 Enterprise SP1 [6.1 Build 7601] (x64)
So it seems the issue might be with the really low random read / write speeds? I have considered disabling the antivirus and testing, however, seeing these values for read and write speeds I don't think the antivirus program can be the reason (since I don't think it would affect random read / write speeds but not sequential ones).

Any ideas what might be causing such speeds, anyone?

Thanks to everyone.

Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.

User avatar
Dalai
Power Member
Power Member
Posts: 6568
Joined: 2005-01-28, 22:17 UTC
Location: Meiningen (Südthüringen)

Re: Slow directory reading

Post by *Dalai » 2019-03-21, 15:16 UTC

CrystalDiskMark doesn't make much sense for a HDD because it mainly benchmarks random I/O where HDD are really really bad at. So, I won't give much about that. What is more important IMO is the inconsistency in HD Tune benchmark. The graph should have less ups and downs. Did you exit any programs and background programs before benchmarking?

And, what's worse: 53°C for is definitely too hot for an HDD! You need to do something about that. That's why GSmartControl also says that it this attribute failed in the past, meaning that it got above some threshold defined by the manufacturer.

Regards
Dalai
#101164 Personal licence
Athlon X4 880K, 16 GiB RAM, Gigabyte F2A88X-D3HP, Win7 x64

Plugins: Services2, Startups

User avatar
Hacker
Moderator
Moderator
Posts: 11213
Joined: 2003-02-06, 14:56 UTC
Location: Bratislava, Slovakia

Re: Slow directory reading

Post by *Hacker » 2019-03-21, 15:41 UTC

Dalai,
CrystalDiskMark doesn't make much sense for a HDD because it mainly benchmarks random I/O where HDD are really really bad at.
I see.
The graph should have less ups and downs. Did you exit any programs and background programs before benchmarking?
No, I tried to perform the benchmark under the usual load conditions. I will try to run it on a system as clean as possible and post the benchmark.
53°C for is definitely too hot for an HDD! You need to do something about that.
There is not much I can do about it, it's in its usual place in the laptop. On one hand there is no special ventilation, on the other hand there is no dust buildup either.

Thank you
Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.

User avatar
makinero
Senior Member
Senior Member
Posts: 268
Joined: 2013-10-26, 10:05 UTC

Re: Slow directory reading

Post by *makinero » 2019-03-21, 15:46 UTC

Temperature : 53 C

Nice oven, the drive will not work so long, and there may be speed problems.

http://www.buildcomputers.net/hdd-temperature.html

AntonDudarenko
Member
Member
Posts: 110
Joined: 2006-11-04, 15:30 UTC
Location: Russian Federation

Re: Slow directory reading

Post by *AntonDudarenko » 2019-03-21, 16:04 UTC

Maybe - by analogy with the above-mentioned self-written program(showfrag) from the author - he should write a test utility for scanning the selected "point" from the file system structure with measurement of the time spent? Using the same functions/approaches/techniques as Total Commander usually does. And so we can immediately find a bottleneck?
it's just that I have never met a situation once, as makinero rightly pointed out above - that some third-party actions with HDD would help to increase the speed of processing by Total Commander folders/files. Thus if Total Commander immediately does something on the disk slowly - then it will always do it slowly - especially with a large number of files and/or directories.
So I just put up with this situation. And if I see that the folder opens for a long time - I just switch to another matter, because already know for sure that instantly this folder will not open((((
#146217 personal license

User avatar
Usher
Senior Member
Senior Member
Posts: 376
Joined: 2011-03-11, 10:11 UTC

Re: Slow directory reading

Post by *Usher » 2019-03-21, 18:54 UTC

2Hacker
This HDD is TOSHIBA MK5061GSYN, 500 GB, 7200 rpm (that's why it's so hot). It's most likely Advanced Format HDD (with 4 KB sectors) running in 512 byte sector emulation mode (partitions are misaligned).
Edit: HDD may also need firmware update, but if it works…
Regards from Poland
Andrzej P. Wozniak

jjk
Member
Member
Posts: 160
Joined: 2003-07-03, 10:41 UTC

Re: Slow directory reading

Post by *jjk » 2019-03-21, 21:00 UTC

And what with an older TC version ? ie TC9 or TC8 ?
Is it particular to TC 9.22 ?

Post Reply