Trid wdx plugin is very slow on processing many files at once

English support forum

Moderators: white, Hacker, petermad, Stefan2

lezerogan2
Junior Member
Junior Member
Posts: 83
Joined: 2010-04-19, 15:50 UTC

Trid wdx plugin is very slow on processing many files at once

Post by *lezerogan2 »

Win 10 home latest 64 bit. TC 10.50 64 bit. I am using the Trid wdx plugin.
I created new columns set. It is working fine. The plugin displays the extension of each file appearing in the current page

But when the current directory contains more than 50 files (more than 1 page), if "page down" is pressed to see the next files, you see at first that the plugin extension column is empty and then one by one the plugin is filling it with the extension value.
If the directory contains hundreds of files, you need to press "page down" and wait 15 sec until this page column is filled, and only then continue to the next page.
I wish that the filling of the column will continue in the background for all files in this directory, while allowing me meanwhile to do other stuff.

Is this behavior is because of TC method or it is the plugin? Can it be solved by some definition in some INI file?
User avatar
Dalai
Power Member
Power Member
Posts: 9365
Joined: 2005-01-28, 22:17 UTC
Location: Meiningen (Südthüringen)

Re: Trid wdx plugin is very slow on processing many files at once

Post by *Dalai »

lezerogan2 wrote: 2022-07-21, 11:04 UTCI am using the Trid wdx plugin.
Which of the two are you using?
https://totalcmd.net/plugring/trid_identifier.html
https://totalcmd.net/plugring/trid.html

The behavior you describe lets me assume that the plugin delivers the data immediately so that everything is running in TC's main/foreground thread.
Is this behavior is because of TC method or it is the plugin?
It's plugin behavior. Nothing TC could do about it.
Can it be solved by some definition in some INI file?
Maybe. Look through the plugin documentation (if any). Check if there's a way to make the plugin tell TC to retrieve the data in a background thread again. But most plugins don't have such setting. There are probably more plugins out there that deliver the data via the background thread (delayed).

Regards
Dalai
#101164 Personal licence
Ryzen 5 2600, 16 GiB RAM, ASUS Prime X370-A, Win7 x64

Plugins: Services2, Startups, CertificateInfo, SignatureInfo, LineBreakInfo - Download-Mirror
User avatar
Hacker
Moderator
Moderator
Posts: 13052
Joined: 2003-02-06, 14:56 UTC
Location: Bratislava, Slovakia

Re: Trid wdx plugin is very slow on processing many files at once

Post by *Hacker »

Dalai,
Dalai wrote: 2022-07-21, 13:17 UTC
lezerogan2 wrote: 2022-07-21, 11:04 UTCIs this behavior is because of TC method or it is the plugin?
It's plugin behavior. Nothing TC could do about it.
Is there a reason why TC cannot request the data for all files in the panel but is restricted to requesting the data for visible files only?

Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.
User avatar
Dalai
Power Member
Power Member
Posts: 9365
Joined: 2005-01-28, 22:17 UTC
Location: Meiningen (Südthüringen)

Re: Trid wdx plugin is very slow on processing many files at once

Post by *Dalai »

Hacker wrote: 2022-07-21, 13:42 UTCIs there a reason why TC cannot request the data for all files in the panel but is restricted to requesting the data for visible files only?
I see several reasons:
  1. If TC would request the data for every file, it would take a very long time for large directories requesting data the user might not even be interested in.
  2. The TC plugin interfaces are written in a way that the communication between TC and a plugin is based on a single file for each call. In other words, a plugin can't return the values for a whole directory at once. It can get the data by itself in advance, and then return that piece by piece. There are situations or plugins where doing this can help, but I doubt it's the majority.
  3. If a plugin isn't implemented thread-safe, TC can't call a plugin interface function from different threads. Doing so would lead to crashes and other unexpected behavior. I've had some first-hand experience with this recently, and I can tell you that you don't want to deal with this as a user ;).
  4. To do something about the previous point, TC would need to request ALL content fields from its background thread. This would create an additional overhead for simple/fast content fields, slowing things down for everyone and every WDX plugin.
BTW, TC does exactly the same for icons and thumbnails, retrieving them from the visible files only.

Regards
Dalai
#101164 Personal licence
Ryzen 5 2600, 16 GiB RAM, ASUS Prime X370-A, Win7 x64

Plugins: Services2, Startups, CertificateInfo, SignatureInfo, LineBreakInfo - Download-Mirror
lezerogan2
Junior Member
Junior Member
Posts: 83
Joined: 2010-04-19, 15:50 UTC

Re: Trid wdx plugin is very slow on processing many files at once

Post by *lezerogan2 »

2Dalai
I am using
https://totalcmd.net/plugring/trid_identifier.html
I did not find documentation.
it is a slow plugin, but it will be fine for me to leave TC for 2 minutes until the plugin will finish to work and fill the column for all files.
I think Hacker phrased the question better than me.
User avatar
Stefan2
Power Member
Power Member
Posts: 4133
Joined: 2007-09-13, 22:20 UTC
Location: Europa

Re: Trid wdx plugin is very slow on processing many files at once

Post by *Stefan2 »

See the screenshots to see that that tool need some time itself to analyses each file:
https://mark0.net/soft-trid-e.html

You may use a script to loop over TCs' temp file (which contains all selected files or with reference to a path\folder)
and call that tool (not the wdx) for each file.
Next there are plugins to execute such script for you and provide column content themself based on the result of each file.
(or let the files be renamed wiht the correct extensions?)




 
Support TrID
If TrID proved useful to you, maybe you can write a comment .....
If you like TrID, you may consider a little donation!

https://mark0.net/soft-trid-e.html
lezerogan2
Junior Member
Junior Member
Posts: 83
Joined: 2010-04-19, 15:50 UTC

Re: Trid wdx plugin is very slow on processing many files at once

Post by *lezerogan2 »

I do not mind about Trid being slow. I can wait.
For other purposes I have already wrote a script calling Trid, that renames files with correct extension.

But there are cases I want to use the wdx inside TC.
From Dalai explanation I understand that because of performance issues, TC will call this plugin only for the visible files.

So for me case is closed. Thank you all for your help.
User avatar
Dalai
Power Member
Power Member
Posts: 9365
Joined: 2005-01-28, 22:17 UTC
Location: Meiningen (Südthüringen)

Re: Trid wdx plugin is very slow on processing many files at once

Post by *Dalai »

OK, so it's the newer one.
I did not find documentation.
It's in the plugin directory.
it is a slow plugin, but it will be fine for me to leave TC for 2 minutes until the plugin will finish to work and fill the column for all files.
Not the plugin itself is slow but the TrID application which takes a while to identify the file type based on its contents. If you happen to have an older version of its ID database (which is smaller), the detection will be faster in my experience.

How many Match columns from this plugin do you have in your custom columns view? IIRC the more such columns there are the longer it will take.

BTW: Both TrID plugins make TC request the data in a background thread. I've just tested it. That means that you can do things while TC gets the data like change directory, scroll through the directory and so on. If this wasn't the case, TC would be unusable (and appear hanging) while it gets the data from the plugin(s).

Regards
Dalai
#101164 Personal licence
Ryzen 5 2600, 16 GiB RAM, ASUS Prime X370-A, Win7 x64

Plugins: Services2, Startups, CertificateInfo, SignatureInfo, LineBreakInfo - Download-Mirror
User avatar
Hacker
Moderator
Moderator
Posts: 13052
Joined: 2003-02-06, 14:56 UTC
Location: Bratislava, Slovakia

Re: Trid wdx plugin is very slow on processing many files at once

Post by *Hacker »

Dalai,
1. If TC would request the data for every file, it would take a very long time for large directories requesting data the user might not even be interested in.
I don't see the disadvantage of preloading the information for files which are not yet visible. Best case - the information is there when I need it and I don't need to wait. Worst case - I don't use the information.
2. The TC plugin interfaces are written in a way that the communication between TC and a plugin is based on a single file for each call. In other words, a plugin can't return the values for a whole directory at once. It can get the data by itself in advance, and then return that piece by piece. There are situations or plugins where doing this can help, but I doubt it's the majority.
In what situation is preloading data not wanted? At least in the case of this plugin I don't see any situation where it would not be beneficial, especially since TrID is not the fastest of programs.
3. If a plugin isn't implemented thread-safe, TC can't call a plugin interface function from different threads. Doing so would lead to crashes and other unexpected behavior. I've had some first-hand experience with this recently, and I can tell you that you don't want to deal with this as a user
I don't see how this is related to the question / request to continue loading information for files further down the list.
4.
See above.

Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.
User avatar
Dalai
Power Member
Power Member
Posts: 9365
Joined: 2005-01-28, 22:17 UTC
Location: Meiningen (Südthüringen)

Re: Trid wdx plugin is very slow on processing many files at once

Post by *Dalai »

2Hacker
You might want to load information for files not visible in the panel, but how would that work in this case where fields take a long time to deliver the data? TC requests the data faster than the plugin can deliver them. As a result, you can't really preload any data. The only way this would work is when a user scrolls through the list slower than the plugin delivers the data. But it's the opposite we're talking about, isn't it?

I don't think it makes a difference for TrID if it detects the type of a single file or that of all files in a directory. The program itself must handle each file separately anyway - and that's what's taking so long. The time required for one file will stay the same (more or less).
Hacker wrote: 2022-07-21, 19:03 UTCWorst case - I don't use the information.
And while doing so you wasted resources. Keep in mind that the information might come from slow drives or slow network connections, or getting it is I/O and/or CPU-intensive; TrID is kind of in the latter category. TC shouldn't do that for every field of every plugin. In any case, it is what it is. Any different behavior would require changes in TC itself and maybe even in the plugin interfaces.

Regards
Dalai
#101164 Personal licence
Ryzen 5 2600, 16 GiB RAM, ASUS Prime X370-A, Win7 x64

Plugins: Services2, Startups, CertificateInfo, SignatureInfo, LineBreakInfo - Download-Mirror
User avatar
Hacker
Moderator
Moderator
Posts: 13052
Joined: 2003-02-06, 14:56 UTC
Location: Bratislava, Slovakia

Re: Trid wdx plugin is very slow on processing many files at once

Post by *Hacker »

Dalai,
I am not sure we're talking about the same thing.

Let's say I have a dir with 100 files having random names, as they have been recovered / undeleted and their original names are lost. 20 are visible per screen and I am looking for a specific PDF.
TC slowly loads the TrID details for the first twenty files, let's say it takes 30 seconds, and I see there is a PDF among them.
I examine it using quickview, to see if it is the file I need. It is not. This took 1 minute.
Now I want to check the next 20 files, and I would have hoped TC already loaded the TrID details for the next 40 files, since it had one minute to do so, but it did not, and I have to wait again 30 seconds for the files on the next screen to load with their TrID details. The process repeats, again, I check one PDF for one minute, it is not the correct one. By this time TC should have loaded the details for all the files, but it does not, and I have to wait again.

I do not see any reason why this would be something beneficial.
And while doing so you wasted resources. Keep in mind that the information might come from slow drives or slow network connections, or getting it is I/O and/or CPU-intensive; TrID is kind of in the latter category.
Please leave it up to me to decide if I rather waste a few CPU cycles or minutes of my time. By the same reasoning we would have to remove the PreloadThumbnails setting because it might load a thumbnail which I might not look at.

Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.
User avatar
Dalai
Power Member
Power Member
Posts: 9365
Joined: 2005-01-28, 22:17 UTC
Location: Meiningen (Südthüringen)

Re: Trid wdx plugin is very slow on processing many files at once

Post by *Dalai »

Hacker wrote: 2022-07-21, 20:01 UTCNow I want to check the next 20 files, and I would have hoped TC already loaded the TrID details for the next 40 files, since it had one minute to do so, but it did not, and I have to wait again 30 seconds for the files on the next screen to load with their TrID details.
That is a good example of where the user is indeed slower than the plugin would be able to deliver the data, and I agree with that. But I didn't say that it couldn't be useful to preload data, but it definitely isn't useful to always do so.

BTW, there's an easy workaround for your example: scroll through the whole directory once. This makes TC request the data of every column for every file. Now I know that this isn't particularly practical for huge directories, so that's why I call it a workaround. Workarounds don't need to be practical ;).

Hacker wrote: 2022-07-21, 20:01 UTCPlease leave it up to me to decide if I rather waste a few CPU cycles or minutes of my time. By the same reasoning we would have to remove the PreloadThumbnails setting because it might load a thumbnail which I might not look at.
That's not what I'm saying. It just depends on the situation, the data and the plugin if it's useful and/or makes sense to preload data. That's all. But you can't just preload everything because that would make things worse for everyone.

Regards
Dalai
#101164 Personal licence
Ryzen 5 2600, 16 GiB RAM, ASUS Prime X370-A, Win7 x64

Plugins: Services2, Startups, CertificateInfo, SignatureInfo, LineBreakInfo - Download-Mirror
User avatar
Hacker
Moderator
Moderator
Posts: 13052
Joined: 2003-02-06, 14:56 UTC
Location: Bratislava, Slovakia

Re: Trid wdx plugin is very slow on processing many files at once

Post by *Hacker »

Dalai,
It just depends on the situation, the data and the plugin if it's useful and/or makes sense to preload data.
Well, it seems at this moment there is no way to set TC to preload content plugin data for all files and I don't see a reason why we do not have the option. For me, if I activate a custom columns view, I'd like to have the data available for all files in the current dir / branch view / etc. as soon as possible. I don't understand the logic behind limiting it to one screen only. It cannot be because of resource wasting, because if my screen is really large, perhaps I already see all 100 files at once, so the details of all 100 get loaded. But if my screen is really small, I have to keep scrolling and scrolling to see all the details because otherwise TC does not load them.
Imagine pressing Alt-Shift-Enter (without Everything) and only the size of the shown dirs would be calculated and you'd have to scroll for TC to start calculating the sizes of the remaining dirs.
I really do not see the logic behind the current behavior. I have always thought there is some .ini setting I must have misconfigured somewhen for this to occur, but it seems this behavior is actually by design.

Christian,
Could you please tell us why it is designed like this and if we could please get an option similar to PreloadThumbnails, something like "PreloadWdxFields" or so, which would load the content plugin data for all files in the current dir, even those which are not visible (i.e. out of the current scroll position)?

TIA
Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.
lezerogan2
Junior Member
Junior Member
Posts: 83
Joined: 2010-04-19, 15:50 UTC

Re: Trid wdx plugin is very slow on processing many files at once

Post by *lezerogan2 »

It seems that the only way we can get info from the plugin for unseen files on screen from current directory, is if Mr. Ghisler will add an option to allow it, and I support that.

2dalai: I have already tried your idea for a workaround, but it does not work.
If TC is in the first page showing 30 files, and the plugin finished to fill only 20 of them, and I press "page down", Tc will jump to the next page, the plugin will start to fill info for the second page, but it will not finished the 10 missing files on the first page, they will remain without info. So this will not work.
User avatar
Dalai
Power Member
Power Member
Posts: 9365
Joined: 2005-01-28, 22:17 UTC
Location: Meiningen (Südthüringen)

Re: Trid wdx plugin is very slow on processing many files at once

Post by *Dalai »

lezerogan2 wrote: 2022-07-22, 07:18 UTCIf TC is in the first page showing 30 files, and the plugin finished to fill only 20 of them, and I press "page down", Tc will jump to the next page, the plugin will start to fill info for the second page, but it will not finished the 10 missing files on the first page, they will remain without info. So this will not work.
It does work, but you have to wait. TC will request data for visible files first, i.e. the last 10 files in your example. After that's done it will retrieve data from the files that were visible previously.

Regards
Dalai
#101164 Personal licence
Ryzen 5 2600, 16 GiB RAM, ASUS Prime X370-A, Win7 x64

Plugins: Services2, Startups, CertificateInfo, SignatureInfo, LineBreakInfo - Download-Mirror
Post Reply