new WDX interface feature contflags_fieldsearch

Discuss and announce Total Commander plugins, addons and other useful tools here, both their usage and their development.

Moderators: white, Hacker, petermad, Stefan2

User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

new WDX interface feature contflags_fieldsearch

Post by *milo1012 »

If this
history1000.txt wrote:01.03.21 Added: Content plugin interface: New flag contflags_fieldsearch allows to handle the search within the plugin instead of returning the value to Total Commander. Includes 3 new functions: ContentFindValue, ContentFindValueW and ContentGetSupportedOperators (32/64)
means what I think it means, we could finally pass sth. like a RegEx directly to plug-ins and let them handle the search with it, instead of just using a pre-configured search/field parameter (like currently in my PCREsearch plug-in).
But even if not: thank you for expanding the wdx interface in a useful way :)

Now the important question: is there already some documentation for the function parameters available?
TC plugins: PCREsearch and RegXtract
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: new WDX interface feature contflags_fieldsearch

Post by *ghisler(Author) »

Yes, that's exactly what it means!

Here is the documentation:
https://www.totalcommander.ch/beta/contentpluginhelp212_chm.zip
Author of Total Commander
https://www.ghisler.com
User avatar
tbeu
Power Member
Power Member
Posts: 1336
Joined: 2003-07-04, 07:52 UTC
Location: Germany
Contact:

Re: new WDX interface feature contflags_fieldsearch

Post by *tbeu »

TC plugins: Autodesk 3ds Max / Inventor / Revit Preview, FileInDir, ImageMetaData (JPG Comment/EXIF/IPTC/XMP), MATLAB MAT-file Viewer, Mover, SetFolderDate, Solid Edge Preview, Zip2Zero and more
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Re: new WDX interface feature contflags_fieldsearch

Post by *milo1012 »

Ok, I had some time to test the new functions with TC 10. Unfortunately they seem to be completely unusable for text field types for now.

Getting ContentGetSupportedFieldFlags() and ContentGetSupportedOperators() to work was no problem. I set a single operator ("regex" or "contains") and mark them as ft_string in ContentGetSupportedField().
Now TC calls ContentFindValue()/ContentFindValueW() with the correct FieldIndex and UnitIndex=0 and OperationIndex=0. FieldType is ft_stringw, which is what I'd expect, since I set it in ContentGetSupportedField() and we're in non-Win9x.

But: FieldValue contains a pointer to garbage data. No matter what I tried: when I fill the corresponding plug-in field in the search dialog's plug-in tab, the pointer seems to point to a wrong or wrongly filled buffer and I get neither a valid ANSI string or Unicode string, not even a valid integer, float, or anything usable. You can even leave the field empty and already get garbage bytes, where I'd expect at least a terminating c-string zero byte. I tried returning ft_stringw and even ft_fulltextw in ContentGetSupportedField() - no difference.

The only good news: a number field seems to work: if I mark the field as ft_numeric_32 or ft_numeric_floating, I can reinterpret the pointer as 32bit integer or double fp number w/o any problems. So it's obvious that the string conversion to the buffer in case of ft_string is completely broken.
BTW for ft_numeric_*- fields: what's the point of letting the user put anything in the plug-In's field value input box that is not a number? This way, the plug-in has no chance to distinguish between a real integer/float of value zero and an invalid input putted in the field (like letters only, or forgetting the comma/period for floating numbers). A message box would be advisable, or at least try to "clean" the string the user put in, when it's obvious that the user tried to put letters into a number field, etc. Not sure about a boolean field. And IMO the plug-in should not be called with the new ContentFindValue at all, when the box is left empty.

Tested 32-bit TC only. I didn't bother testing x64, since it's obvious that there's something completely wrong and for me the new functions are unusable so far. I need to get a clean text input.

And another bug with the new function: in the "new" (TC 9 new) wdx plug-in field picker dialog ("Content field chooser dialog"): when I open it on the "Find files: General" dialog via "Find text" -> Plugins: when jumping to a field configured with contflags_fieldsearch, TC seems to call it with FieldType set to 2048 (0x800), which is not a valid type. flags was set to CONTENT_DELAYIFSLOW in this case.
TC plugins: PCREsearch and RegXtract
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: new WDX interface feature contflags_fieldsearch

Post by *ghisler(Author) »

Sorry for the confusion. When searching for ft_string or ft_stringw, the FieldValue parameter isn't directly a string. Instead it contains one or more pointers to zero-terminated strings (char*, WCHAR*):
ft_string: FieldValue contains a pointer to a zero-terminated ANSI string
ft_stringw: FieldValue contains:
- a pointer to a zero-terminated ANSI string
- a pointer to a zero-terminated Unicode UTF-16 string
- a 32-bit BOOL value which is not equal to 0 when the Unicode string contains characters which are lost when converted to ANSI
Author of Total Commander
https://www.ghisler.com
User avatar
nsp
Power Member
Power Member
Posts: 1803
Joined: 2005-12-04, 08:39 UTC
Location: Lyon (FRANCE)
Contact:

Re: new WDX interface feature contflags_fieldsearch

Post by *nsp »

ghisler(Author) wrote: 2022-01-28, 10:51 UTC Sorry for the confusion. When searching for ft_string or ft_stringw, the FieldValue parameter isn't directly a string. Instead it contains one or more pointers to zero-terminated strings (char*, WCHAR*):
ft_string: FieldValue contains a pointer to a zero-terminated ANSI string
ft_stringw: FieldValue contains:
- a pointer to a zero-terminated ANSI string
- a pointer to a zero-terminated Unicode UTF-16 string
- a 32-bit BOOL value which is not equal to 0 when the Unicode string contains characters which are lost when converted to ANSI
@ghisler(Author)
Christian, do you have a reference implementation in C for this new feature ?


Imagine i have an ini parser wdx, i want to verify that the attribute MySection.myValue is set or have a defined Value.
Can i do a searchable string field isSet use operator contain and type a value MySection.myValue
or do a fulltext search on which i can define a search grammar "MySection.myValue is set" "MySection.myValue = ValueToCheck" ...
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Re: new WDX interface feature contflags_fieldsearch

Post by *milo1012 »

ghisler(Author) wrote: 2022-01-28, 10:51 UTCWhen searching for ft_string or ft_stringw, the FieldValue parameter isn't directly a string. Instead it contains one or more pointers to zero-terminated strings (char*, WCHAR*)
I see. Thanks for the clarification! I'll try that as soon as possible. I wonder if a double pointer would have been better for this(void**), as it would make things clear just by looking at the function's signature. (sure, this wouldn't work for the non-sting field types).
In fact, https://ghisler.github.io/WDX-SDK/hs2200.htm says this, but I probably missed the commas in "ft_stringw,ft_fulltext, ft_fulltextw:".
ghisler(Author) wrote: 2022-01-28, 10:51 UTCft_string: FieldValue contains a pointer to a zero-terminated ANSI string
But wouldn't the "recoding" information also be useful for ft_string? If I enter Unicode chars et.al in the text box, which have no direct mapping in the current ANSI page, a replacement character will compensate this (usually the question mark). There's no way to know if the user entered them directly or if they got there by recoding. But sure, most users will prob. use ft_stringw anyway.

There's still that strange behavior with FieldType set to 0x800, though.
TC plugins: PCREsearch and RegXtract
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Re: new WDX interface feature contflags_fieldsearch

Post by *milo1012 »

Alright, I did some further tests.
First of all, https://ghisler.github.io/WDX-SDK/contents.htm seems to have some information missing from https://www.totalcommander.ch/beta/contentpluginhelp212_chm.zip
E.g https://ghisler.github.io/WDX-SDK/hs2200.htm seems to have cut off the last "Note about Unicode". That's why I read over that requesting ft_string for a field would never actually return a simply ANSI string, but ft_stringw in ContentFindValue in all cases.

So for the tests: the "2 separate pointer mechanism" seems to work, I get an ANSI and Unicode string. But the BOOL flag following the Unicode string is unclear to me. I interpret the next 4 bytes after the terminating double-zero byte for the Unicode string as a BOOL. But it does not seem be a true Windows "BOOL", which is defined as a C/C++ "int" (i.e. is 32 bit in x86 and 64 due to LLP64): I get either a "true" value, no matter what string I enter, or it's not supposed to be 32-bit long. E.g. I enter some test strings and this is what I interpret in ContentFindValue:

Code: Select all

abc
abc
0x16

abc def
abc def
0x18

abc ?ef
abc ḋef
0x108e
All of these last BOOL values in the 3 examples would define as "true". So could you please elaborate this to us?

As for the strange behavior for ContentFindValue and FieldType set to 0x800: at the same call, FieldValue is set to en empty pointer/null, whenever ContentFindValue is called from the "Choose plugin property" dialog box. So should we indicate a non-valid FieldType (0x800) as a "dummy call", i.e. from this chooser dialog? I guess we need at least some kind of mechanism, because: If the user keeps an empty string in the find files plug-in tab, FieldValue is a valid double pointer to empty strings. Wouldn't it better for TC to not call ContentFindValue in the chooser dialog, or not even show fields flagged with contflags_fieldsearch?
In any case, this should be documented.
TC plugins: PCREsearch and RegXtract
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: new WDX interface feature contflags_fieldsearch

Post by *ghisler(Author) »

I just noticed that I didn't publish the new content plugin help, although I finished it in April 2021. :(
Here it is:
https://ghisler.fileburst.com/content/contentpluginhelp212_chm.zip
So for the tests: the "2 separate pointer mechanism" seems to work, I get an ANSI and Unicode string. But the BOOL flag following the Unicode string is unclear to me.
That's odd, it is set either to 00 00 00 00 for false or to FF FF FF FF for true. The 4 bytes directly follow the two pointers. Maybe an alignment problem?
And another bug with the new function: in the "new" (TC 9 new) wdx plug-in field picker dialog ("Content field chooser dialog"): when I open it on the "Find files: General" dialog via "Find text" -> Plugins: when jumping to a field configured with contflags_fieldsearch, TC seems to call it with FieldType set to 2048 (0x800), which is not a valid type. flags was set to CONTENT_DELAYIFSLOW in this case.
First, I cannot reproduce that - my own internal plugin (ContentFindValue/W used for tc->partner file with other extension) is never called when I add that field to the text search function.
Second, you cannot use these fields here. There is a "!" shown in front of the field to warn the user. I think I should show an error, or not show the field at all. The reason is how this function works: You can add multiple fields, TC will load them all and then search the resulting text for the specified search string.
So I think the best solution would be to return an error like ft_notsupported when you encounter invalid parameters.
Author of Total Commander
https://www.ghisler.com
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Re: new WDX interface feature contflags_fieldsearch

Post by *milo1012 »

ghisler(Author) wrote: 2022-01-31, 11:02 UTCThat's odd, it is set either to 00 00 00 00 for false or to FF FF FF FF for true. The 4 bytes directly follow the two pointers. Maybe an alignment problem?
Erm, the docs say:
FieldValue is a pointer to 2 separate pointers: The first points to a 0 terminated ANSI string, the second to a 0 terminated Unicode string. Behind it follows a BOOL value telling the user whether the Unicode string contains any characters which cannot be represented by the ANSI string.
That's why I said and tried:
I interpret the next 4 bytes after the terminating double-zero byte for the Unicode string as a BOOL
So I will try a BOOL following the Unicode string pointer, and not following the string itself. This description is really confusing.
So let me get this straight: FieldValue is a pointer to a pointer, followed by a pointer to a pointer, followed by a Windows BOOL (aka 32-bit int)?
ghisler(Author) wrote: 2022-01-31, 11:02 UTCSecond, you cannot use these fields here. There is a "!" shown in front of the field to warn the user.
I don't see such "!" for the fields flagged with contflags_fieldsearch. I have a mixed field set: some w/o contflags_fieldsearch, some with it. And for those with the flag, TC calls ContentFindValue(W) in this dialog like I described.
And yes, I didn't expect them to be usable from here, I just wanted to check if they are listed here, which they are, and I didn't expect a call to ContentFindValue from here at all.

I'm trying to provide a sample plug-in with my mixed field examples, when I have the time for it. But I think it would be better to update the official sample plug-in (https://ghisler.fileburst.com/content/wdx_filesys_src.zip) with a working example for ContentFindValue.
ghisler(Author) wrote: 2022-01-31, 11:02 UTCSo I think the best solution would be to return an error like ft_notsupported when you encounter invalid parameters.
Ok, but it still feels like this is a wrong behavior from TC side, since everything else seems to work so far.
TC plugins: PCREsearch and RegXtract
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: new WDX interface feature contflags_fieldsearch

Post by *ghisler(Author) »

So let me get this straight: FieldValue is a pointer to a pointer, followed by a pointer to a pointer, followed by a Windows BOOL (aka 32-bit int)?
Yes, exactly.
Ok, but it still feels like this is a wrong behavior from TC side, since everything else seems to work so far.
How should it handle this for file content when the user has specified multiple plugin fields?
Author of Total Commander
https://www.ghisler.com
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Re: new WDX interface feature contflags_fieldsearch

Post by *milo1012 »

ghisler(Author) wrote: 2022-01-31, 14:45 UTCHow should it handle this for file content when the user has specified multiple plugin fields?
I meant that a certain combination of invalid parameters is not what I'd except to check in the plug-in itself. Checking pointers for null values (FieldValue): of course, checking a invalid field number: sure. But a non-valid field type?
But anyway, I'll see that I add some more checks, and maybe there's another reason why I don't see the "!".
TC plugins: PCREsearch and RegXtract
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Re: new WDX interface feature contflags_fieldsearch

Post by *milo1012 »

Ok, here's a simple q&d sample plug-in with C++ source:
http://wincmd.ru/files/9924355/tc10_wdx_test.zip

The 0x800 came through another misunderstanding: the docs clearly say:
contflags_fieldsearch
Pass the search string to the plugin via ContentFindValue instead of calling ContentGetValue and comparing the results. New in Total Commander 10.
As I didn't expect a call to ContentGetValue(), I used the maxlen parameter for FieldType in an internal call for a function handling both ContentFindValue() and ContentGetValue().
Sorry for the confusion. So all in all this means that TC makes a call to ContentGetValue() for a field flagged with contflags_fieldsearch. I call a messag box in the plug-in for this case.

So why does TC call ContentGetValue() in the plug-in chooser dialog despite the flag and the docs? This is the most confusing part: sure, having a field handling both functions might be useful at some point, but this isn't consistent with the other wdx features: if I mark a field for "compare by content", I'm expecting it to be only requested from ContentCompareFiles(), and so on.
It's even more confusing as the same happens with the custom columns editor dialog: TC calls ContentGetValue() when I click on such field in the dialog. Same with the custom column view itself: when I've selected such field for a view, TC calls ContentGetValue() for each file.

And concerning the "!": this will only show if I combine contflags_fieldsearch with contflags_searchpageonly. But even then TC calls ContentGetValue() in the plug-in chooser dialog from via "Find text" -> Plugins, but they seem to not appear in the custom column chooser dialog at all.


In any case: I will use contflags_fieldsearch only in combination with contflags_searchpageonly. And I can confirm that the double pointer/BOOL mechanism works, though it's quite cumbersome.
TC plugins: PCREsearch and RegXtract
User avatar
nsp
Power Member
Power Member
Posts: 1803
Joined: 2005-12-04, 08:39 UTC
Location: Lyon (FRANCE)
Contact:

Re: new WDX interface feature contflags_fieldsearch

Post by *nsp »

To me it seems that the contflags_fieldsearch make sense only with ft_fulltext(w) as it can only return a boolean filling the BOOL pointed value reading the search term and the operator.
It make to me no sense to return a value globally to be seen in a custom column or whatever and to be the search result if it only returns TRUE or FALSE.


What seems to not be supported is extracting/searching and then returning a value (numeric or string). Let say i search for a word and want to return the number of occurrences number of lines containing it.... The same for an ini parser wdx searching for a "searchable" property and returning the value. THis is why for me current implemetation should only target fulltext mechanism as it is already protected against other application than search.
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: new WDX interface feature contflags_fieldsearch

Post by *ghisler(Author) »

So why does TC call ContentGetValue() in the plug-in chooser dialog despite the flag and the docs?
It doesn't do this here with my internal field tc->partner file with other extension, so it must be caused by specific parameters.
If you can send me a pre-release version of your plugin, I can check in the debugger why it happens.
To me it seems that the contflags_fieldsearch make sense only with ft_fulltext(w) as it can only return a boolean filling the BOOL pointed value reading the search term and the operator.
Have a look at my internal field tc->partner file with other extension in TC 10. It's used for searching. For example, seearch for
*.cr2 and set tc->partner file with other extension to jpg to find cr2 raw images which have a partner file with the same name but extension jpg.
Author of Total Commander
https://www.ghisler.com
Post Reply