Slow directory reading
Moderators: white, Hacker, petermad, Stefan2
Slow directory reading
Hi,
TC 9.22 x64 on Windows 7 x64. Old laptop with a 5400 RPM 7200 RPM hard drive (not sleeping).
I have an inactive tab which points to a directory with ~2500 files and is sorted by date. I have two color filters, one for NTFS compressed files and one for files which contain any descript.ion comment (tc comment regex .). None of the files match the color filter criteria, if it's relevant in any way.
Now when I click the tab, it takes about 5 seconds 20 seconds to show the files. For comparison, Everything shows all ~700 thousand files on my hard drive sorted by date instantaneously (no color filters of course).
Is there any way to make reading and showing the contents of this tab faster (save for buying a faster hard drive / SSD)?
TIA
Roman
TC 9.22 x64 on Windows 7 x64. Old laptop with a 5400 RPM 7200 RPM hard drive (not sleeping).
I have an inactive tab which points to a directory with ~2500 files and is sorted by date. I have two color filters, one for NTFS compressed files and one for files which contain any descript.ion comment (tc comment regex .). None of the files match the color filter criteria, if it's relevant in any way.
Now when I click the tab, it takes about 5 seconds 20 seconds to show the files. For comparison, Everything shows all ~700 thousand files on my hard drive sorted by date instantaneously (no color filters of course).
Is there any way to make reading and showing the contents of this tab faster (save for buying a faster hard drive / SSD)?
TIA
Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.
- ghisler(Author)
- Site Admin
- Posts: 48124
- Joined: 2003-02-04, 09:46 UTC
- Location: Switzerland
- Contact:
Re: Slow directory reading
Try disk defragmentation via right click on the drive letter in TC - Properties.
Author of Total Commander
https://www.ghisler.com
https://www.ghisler.com
Re: Slow directory reading
Christian,
Thank you. The analysis says the disk is 0% fragmented but I will give it a try anyways. Until then, any other ideas I might try as well?
TIA
Roman
Thank you. The analysis says the disk is 0% fragmented but I will give it a try anyways. Until then, any other ideas I might try as well?
TIA
Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.
Re: Slow directory reading
Turn off icons and compare results.
Andrzej P. Wozniak
Polish subforum moderator
Polish subforum moderator
Re: Slow directory reading
Usher,
Hmm, good idea, but all files are *.txt files so that shouldn't be a problem. I'll try anyways.
Also it seems I have underestimated the read time by quite a lot in my original post, since after defragmentation it is ~20 seconds for a directory of ~2500 *.txt files (sorted by date, no matching color filters) and ~5 seconds for a directory of ~700 *.txt files (same criteria as the previous dir).
Roman
Hmm, good idea, but all files are *.txt files so that shouldn't be a problem. I'll try anyways.
Also it seems I have underestimated the read time by quite a lot in my original post, since after defragmentation it is ~20 seconds for a directory of ~2500 *.txt files (sorted by date, no matching color filters) and ~5 seconds for a directory of ~700 *.txt files (same criteria as the previous dir).
Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.
Re: Slow directory reading
Is it possible that the disk has trouble reading sectors or is even dying? Take a look at the SMART values with CrystalDiskInfo and maybe check the (theoretical) performance with HD Tune.
Regards
Dalai
Regards
Dalai
#101164 Personal licence
Ryzen 5 2600, 16 GiB RAM, ASUS Prime X370-A, Win7 x64
Plugins: Services2, Startups, CertificateInfo, SignatureInfo, LineBreakInfo - Download-Mirror
Ryzen 5 2600, 16 GiB RAM, ASUS Prime X370-A, Win7 x64
Plugins: Services2, Startups, CertificateInfo, SignatureInfo, LineBreakInfo - Download-Mirror
- ghisler(Author)
- Site Admin
- Posts: 48124
- Joined: 2003-02-04, 09:46 UTC
- Location: Switzerland
- Contact:
Re: Slow directory reading
These are extremely slow read times even for a hdd!
It's possible that defrag didn't do anything to the directory because the fragmentation level was too small.
Here is a small tool I wrote in 2006 which can show the number of fragments of a file or directory, and their location on disk:
https://www.totalcommander.ch/win/showfrag.zip
I have signed it with my certificates now to show that it's authentic.
It's possible that defrag didn't do anything to the directory because the fragmentation level was too small.
Here is a small tool I wrote in 2006 which can show the number of fragments of a file or directory, and their location on disk:
https://www.totalcommander.ch/win/showfrag.zip
I have signed it with my certificates now to show that it's authentic.
Author of Total Commander
https://www.ghisler.com
https://www.ghisler.com
Re: Slow directory reading
I have already written about it many times. TC is not adapted to read large directories, and in particular with small files, where speed drops drastically.
I know from experience that people try various ways to improve speed a) defragmentation b) check SMART
But is real? It will not help, the speed will not improve. Although you can try, and the results are always deplorable.
Disable showing icons in TC, should in some ways help, but all in all: There are tools whose icons do not affect speed.
I know from experience that people try various ways to improve speed a) defragmentation b) check SMART
But is real? It will not help, the speed will not improve. Although you can try, and the results are always deplorable.
Disable showing icons in TC, should in some ways help, but all in all: There are tools whose icons do not affect speed.
Re: Slow directory reading
Christian,
Thank you for showfrag. For both the 20 seconds dir with ~2500 files and the 5 seconds dir with ~700 files it shows "Number of fragments: 1".
makinero,
I do not really consider a dir with ~2500 files especially large.
Dalai,
CrystalDiskInfo shows the health as "Good":
GSMartControl also shows "Passed":
HD Tune looks more or less OK, I guess?
However, CrystalDiskMark results look abysmal:
So it seems the issue might be with the really low random read / write speeds? I have considered disabling the antivirus and testing, however, seeing these values for read and write speeds I don't think the antivirus program can be the reason (since I don't think it would affect random read / write speeds but not sequential ones).
Any ideas what might be causing such speeds, anyone?
Thanks to everyone.
Roman
Thank you for showfrag. For both the 20 seconds dir with ~2500 files and the 5 seconds dir with ~700 files it shows "Number of fragments: 1".
makinero,
I do not really consider a dir with ~2500 files especially large.
Dalai,
CrystalDiskInfo shows the health as "Good":
Code: Select all
----------------------------------------------------------------------------
CrystalDiskInfo 8.0.0 (C) 2008-2018 hiyohiyo
Crystal Dew World : https://crystalmark.info/
----------------------------------------------------------------------------
OS : Windows 7 Enterprise SP1 [6.1 Build 7601] (x64)
Date : 2019/03/21 11:40:27
----------------------------------------------------------------------------
(1) TOSHIBA MK5061GSYN
----------------------------------------------------------------------------
Model : TOSHIBA MK5061GSYN
Firmware : MH000C
Disk Size : 500,1 GB (8,4/137,4/500,1/500,1)
Buffer Size : 16384 KB
Queue Depth : 32
# of Sectors : 976773168
Rotation Rate : 7200 RPM
Interface : Serial ATA
Major Version : ATA8-ACS
Minor Version : ----
Transfer Mode : SATA/300 | SATA/300
Power On Hours : 11736 hours
Power On Count : 1525 count
Temperature : 53 C (127 F)
Health Status : Good
Features : S.M.A.R.T., APM, 48bit LBA, NCQ
APM Level : 0080h [ON]
AAM Level : ----
Drive Letter : C:
-- S.M.A.R.T. --------------------------------------------------------------
ID Cur Wor Thr RawValues(6) Attribute Name
01 100 100 _50 000000000000 Read Error Rate
02 100 100 _50 000000000000 Throughput Performance
03 100 100 __2 0000000008DF Spin-Up Time
04 100 100 __0 00000000061E Start/Stop Count
05 100 100 _10 000000000000 Reallocated Sectors Count
07 100 100 _50 000000000000 Seek Error Rate
08 100 100 _50 000000000000 Seek Time Performance
09 _71 _71 __0 000000002DD8 Power-On Hours
0A 131 100 _30 000000000000 Spin Retry Count
0C 100 100 __0 0000000005F5 Power Cycle Count
B7 100 100 __1 000000000003 Vendor Specific
B8 100 100 _97 000000000000 End-to-End Error
B9 100 100 __1 00000000FFFF Vendor Specific
BB 100 100 __0 000000000000 Reported Uncorrectable Errors
BC 100 _99 __0 000000000028 Command Timeout
BD 100 100 __1 000000000000 High Fly Writes
BE _47 _36 _40 002035150035 Airflow Temperature
BF 100 100 __0 000000000012 G-Sense Error Rate
C0 100 100 __0 000000350035 Power-off Retract Count
C1 _96 _96 __0 00000000A6ED Load/Unload Cycle Count
C4 100 100 __0 000000000000 Reallocation Event Count
C5 100 100 __0 000000000000 Current Pending Sector Count
C7 200 200 __0 000000000000 UltraDMA CRC Error Count
Code: Select all
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 100 100 050 - 0
2 Throughput_Performance POS--K 100 100 050 - 0
3 Spin_Up_Time PO---K 100 100 002 - 2271
4 Start_Stop_Count -O--CK 100 100 000 - 1566
5 Reallocated_Sector_Ct PO--CK 100 100 010 - 0
7 Seek_Error_Rate POSR-K 100 100 050 - 0
8 Seek_Time_Performance P-S--K 100 100 050 - 0
9 Power_On_Minutes -O--CK 071 071 000 - 195h+37m
10 Spin_Retry_Count PO--CK 131 100 030 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 1525
183 Runtime_Bad_Block -O--CK 100 100 001 - 3
184 End-to-End_Error PO--CK 100 100 097 - 0
185 Unknown_Attribute -O--CK 100 100 001 - 65535
187 Reported_Uncorrect -O--CK 100 100 000 - 0
188 Command_Timeout -O--CK 100 099 000 - 40
189 High_Fly_Writes -O-RCK 100 100 001 - 0
190 Airflow_Temperature_Cel -O---K 048 036 040 Past 52 (Min/Max 21/53 #32)
191 G-Sense_Error_Rate -O--CK 100 100 000 - 18
192 Power-Off_Retract_Count -O---K 100 100 000 - 3473461
193 Load_Cycle_Count -O--CK 096 096 000 - 42733
196 Reallocated_Event_Count -O--CK 100 100 000 - 0
197 Current_Pending_Sector -O--CK 100 100 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
However, CrystalDiskMark results look abysmal:
Code: Select all
-----------------------------------------------------------------------
CrystalDiskMark 6.0.2 x64 (C) 2007-2018 hiyohiyo
Crystal Dew World : https://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes
Sequential Read (Q= 32,T= 1) : 57.773 MB/s
Sequential Write (Q= 32,T= 1) : 58.303 MB/s
Random Read 4KiB (Q= 8,T= 8) : 0.315 MB/s [ 76.9 IOPS]
Random Write 4KiB (Q= 8,T= 8) : 0.684 MB/s [ 167.0 IOPS]
Random Read 4KiB (Q= 32,T= 1) : 0.323 MB/s [ 78.9 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 0.737 MB/s [ 179.9 IOPS]
Random Read 4KiB (Q= 1,T= 1) : 0.298 MB/s [ 72.8 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 0.811 MB/s [ 198.0 IOPS]
Test : 1024 MiB [C: 96.6% (449.9/465.7 GiB)] (x5) [Interval=5 sec]
Date : 2019/03/21 11:38:18
OS : Windows 7 Enterprise SP1 [6.1 Build 7601] (x64)
Any ideas what might be causing such speeds, anyone?
Thanks to everyone.
Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.
Re: Slow directory reading
CrystalDiskMark doesn't make much sense for a HDD because it mainly benchmarks random I/O where HDD are really really bad at. So, I won't give much about that. What is more important IMO is the inconsistency in HD Tune benchmark. The graph should have less ups and downs. Did you exit any programs and background programs before benchmarking?
And, what's worse: 53°C for is definitely too hot for an HDD! You need to do something about that. That's why GSmartControl also says that it this attribute failed in the past, meaning that it got above some threshold defined by the manufacturer.
Regards
Dalai
And, what's worse: 53°C for is definitely too hot for an HDD! You need to do something about that. That's why GSmartControl also says that it this attribute failed in the past, meaning that it got above some threshold defined by the manufacturer.
Regards
Dalai
#101164 Personal licence
Ryzen 5 2600, 16 GiB RAM, ASUS Prime X370-A, Win7 x64
Plugins: Services2, Startups, CertificateInfo, SignatureInfo, LineBreakInfo - Download-Mirror
Ryzen 5 2600, 16 GiB RAM, ASUS Prime X370-A, Win7 x64
Plugins: Services2, Startups, CertificateInfo, SignatureInfo, LineBreakInfo - Download-Mirror
Re: Slow directory reading
Dalai,
Thank you
Roman
I see.CrystalDiskMark doesn't make much sense for a HDD because it mainly benchmarks random I/O where HDD are really really bad at.
No, I tried to perform the benchmark under the usual load conditions. I will try to run it on a system as clean as possible and post the benchmark.The graph should have less ups and downs. Did you exit any programs and background programs before benchmarking?
There is not much I can do about it, it's in its usual place in the laptop. On one hand there is no special ventilation, on the other hand there is no dust buildup either.53°C for is definitely too hot for an HDD! You need to do something about that.
Thank you
Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.
Re: Slow directory reading
Temperature : 53 C
Nice oven, the drive will not work so long, and there may be speed problems.
http://www.buildcomputers.net/hdd-temperature.html
Nice oven, the drive will not work so long, and there may be speed problems.
http://www.buildcomputers.net/hdd-temperature.html
Re: Slow directory reading
Maybe - by analogy with the above-mentioned self-written program(showfrag) from the author - he should write a test utility for scanning the selected "point" from the file system structure with measurement of the time spent? Using the same functions/approaches/techniques as Total Commander usually does. And so we can immediately find a bottleneck?
it's just that I have never met a situation once, as makinero rightly pointed out above - that some third-party actions with HDD would help to increase the speed of processing by Total Commander folders/files. Thus if Total Commander immediately does something on the disk slowly - then it will always do it slowly - especially with a large number of files and/or directories.
So I just put up with this situation. And if I see that the folder opens for a long time - I just switch to another matter, because already know for sure that instantly this folder will not open((((
it's just that I have never met a situation once, as makinero rightly pointed out above - that some third-party actions with HDD would help to increase the speed of processing by Total Commander folders/files. Thus if Total Commander immediately does something on the disk slowly - then it will always do it slowly - especially with a large number of files and/or directories.
So I just put up with this situation. And if I see that the folder opens for a long time - I just switch to another matter, because already know for sure that instantly this folder will not open((((
#146217 personal license
Re: Slow directory reading
2Hacker
This HDD is TOSHIBA MK5061GSYN, 500 GB, 7200 rpm (that's why it's so hot). It's most likely Advanced Format HDD (with 4 KB sectors) running in 512 byte sector emulation mode (partitions are misaligned).
Edit: HDD may also need firmware update, but if it works…
This HDD is TOSHIBA MK5061GSYN, 500 GB, 7200 rpm (that's why it's so hot). It's most likely Advanced Format HDD (with 4 KB sectors) running in 512 byte sector emulation mode (partitions are misaligned).
Edit: HDD may also need firmware update, but if it works…
Andrzej P. Wozniak
Polish subforum moderator
Polish subforum moderator
Re: Slow directory reading
And what with an older TC version ? ie TC9 or TC8 ?
Is it particular to TC 9.22 ?
Is it particular to TC 9.22 ?