Everything integration in TC

[Son_Of_Diablo]

Hello,


I'm trying to figure out how to use Everything to search through TC.

I have been doing some Google research and read a few threads here on the Forum and so far my understanding of how it works is this:

I need to have Everything installed and running as a service.
I need to download the Everything dlls from the SDK page and place either the 32 or 64 bit dll in the TC Home directory.
I need to have drive/path I'm trying to search indexed in Everything.
When searching in TC (Alt+F7/The "Find Files" Window) I need to check the 'Everything' box.
In my search query (the "Search for" box) I have to start with either "ed:" or "ev:".
"ed:" will respect my choices in "Search in" while "ev:" will ignore those choices.

And... Well, after playing around with this for a little bit I have found the integration to be somewhat faulty.

Sometimes when I search it works as expected and I get a very fast response with the desired items.

However other times (even with the exact same parameters) I either get a bunch of random results or none at all.
It also seems that at times TC doesn't even use Everything and just uses it's build-in search function instead.


So well, my question is this: Have I understood how to use the integration correctly?

And a few follow ups:
In Everything there is this system with Macros/Filters, I have tried using the video filter in TC but I haven't been able to get anything back that was even remotely correct. I tried with the date modified filter (dm:today) and that seemed to work (when TC actually returned a result). are the custom filters not supported by the integration?

Does TC find the running instance of Everything or should I make sure Everything is only running as a service?

I have noticed that TC sometimes display a pop-up window saying something along the lines of "The main Everything window is busy. Please wait" when you check the Everything box in the TC "Find files" window, should I be concerned about this or just let it do it's thing?


I'm currently running with TC 9.22a (x64) and Everything 1.4.1.935 (x64). And I have placed the Everything64.dll from the SDK into the TC Home Directory (Same folder as the TOTALCMD64.EXE).


Any help towards tracking down the problem would be greatly appreciated!

[petermad]

I need to download the Everything dlls from the SDK page and place either the 32 or 64 bit dll in the TC Home directory.

That is not necesscary since TC 9.0rc4
history.txt:

30.10.16 Fixed: Everything dlls are no longer needed, replaced by internal code calling the Everything service directly. Necessary because the dlls didn't support the new Everything 1.4 functions yet (32/64)

.

In my search query (the "Search for" box) I have to start with either "ed:" or "ev:".

That is only necessary if you want 'Everything' to search only in specified directories, or you want to use 'Everytings's search syntax.


My 'Everything' setup:
https://madsenworld.dk/tcmd/everything1.png
https://madsenworld.dk/tcmd/everything2.png
https://madsenworld.dk/tcmd/everything3.png
https://madsenworld.dk/tcmd/everything4.png
https://madsenworld.dk/tcmd/everything5.png

If your drive is not an NTFS drive you have to add it as a folder:
https://madsenworld.dk/tcmd/everything6.png

[Son_Of_Diablo]

2petermad

My settings are the same as yours (at least the highlighted parts).

An example of what I'm encountering is the following:

Image: https://i.imgur.com/i4fsupP.png

The drives are NTFS and the shares are added as folders in Everything and indexed. (I can get results from them in the Everything GUI)

The result from TC:
Image: https://i.imgur.com/oLztSjM.png

Somehow it managed to find 116 folders (Which I find odd given the ext:mkv filter)

Everything only finds 7786 total items:
Image: https://i.imgur.com/Ve5rZFH.png


So what exactly is going wrong?


I have also been playing around with TC after making the post originally and I have noticed that when a search query actually works it takes a few "Searches", almost as if TC is loading results in batches and first displaying the correct result when it has received/processed all batches.

[petermad]

Try in 'Everything's Options under Indexes to use the "Force Rebuild" button, and under Folders to use the "Rescan All Now" button to see if that helps.

How does the search go for you if you remove the two shares ?

[Son_Of_Diablo]

I tried only local drives (before rebuild/rescan), with the same query, first time I hit search it went through a number of directories (as indicated by the status bar at the bottom of the search window) and returned nothing. However second time I hit search (not a new search) it returned the correct result. (Again, it seems like TC uses the first, or few first, searches to pull an index or something).

After the rebuild/rescan, searching just the local drives gives the same result:
First search takes a while and returns nothing (or some random assortment of files and folders), second search returns the expected result.

Adding the shares to the "Search in" gives some odd results, the first search returns nothing (however it takes longer for TC to search files on the shares, feels like it might not use the Everything index?) and it took me about 4 searches after that, each returning a seemingly random mix of files and folders, before I get the expected result. (again taking longer with getting results from the shares).

While if I search in Everything I get what I would expect to get right away, no delay, with just local drives or including the shares.

(Also of note, the "video:" macro still doesn't work in TC, not even with just the local drives)


Edit:

If I hit Search again after getting the correct result with TC I seem to be getting the random mix of folders and files again for a while before I hit the correct result.

[petermad]

In fear of suggesting the obvious - have you tried a restart of you computer...?

[Son_Of_Diablo]

I have indeed.
I guess it's also worth noting that I have had both TC and Everything installed for years, but never used them together until now.

Edit:
I restarted my PC again (anyways) and yes the behavior is still the same.

[petermad]

Could you try with a clean wincmd.ini file?

[Son_Of_Diablo]

I just tried renaming my old wincmd.ini and start TC, and it's behaving the same :/


Edit:

Also it seems that when TC is returning nothing, the search is also longer. I was watching the status bar and it looks like it was running a normal TC search, which would explain why it's returning nothing at all.
However I still haven't been able to pickup any pattern when it's returning random files and folders

[NotNull]

I have also added a WINCMD.ini entry:

Code: Select all

[Configuration]
Everything=c:\tools\everything\Everything.exe

That will cause Everything to be started if not running (when Everything is enabled for a search).
But that is not relevant for the current issue...


Some other suggestions:
Before starting your search, start Everything (the GUI) and press CTRL+`
That will open an Everything debug console.
Start your search in TC
Search for something similar to this (starting with IPC) in the debug console:

Code: Select all

IPC: execute query: path:"c:\program files\"  ext:exe;avi

That will tell you what search query Everything receives from Total Commander. That should help to zoom in where in the chain this goes wrong.
To close the debug window, press CTRL+`again in the Everything GUI (don't use the [x] in the console window!)

If you try the latest 9.50 beta release, it will show you in the statusbar which command is send to Everything.
(see this thread)
Maybe that will give you some extra information.

BTW: I like your Everything colour scheme

[Son_Of_Diablo]

Debug from running (https://pastebin.com/dPUab721):

Code: Select all

IPC: execute query: path:"c:\program files\"  ext:exe;avi

Debug from running (https://pastebin.com/4pW4Hd6J):

Code: Select all

IPC: execute query: path:"c:\|d:\|e:\"  ext:mkv

I'm not sure exactly what to look for in those...
I did notice that TC sends whats in "Search in" and if it's empty it's sending the directory I opened "Find Files" from.
Also I didn't get any results in TC, but I don't know if that is because of the command format?


I played around with it a bit and found that you didn't need "IPC: execute query:" to see what TC is sending.. Unless I missed something important.

So I found a few things:

• If the TC search stops early TC won't disconnect correctly from Everything and the remaining search results will be included in your next search.
• If I searched for something that yielded no actual results, either TC or Everything would be confused and random files/folders would be included in the next search (I searched 'ed: ext:mp3' followed by 'ed: ext:mkv' here is the output: https://pastebin.com/uwyCG1T5)
• The slowness I noticed (that I wrongly attributed to TC doing a normal search instead of a Everything query) seems to appear when the search returns a large amount of results. If I understand the output (at least a little) it seems that the search happens in stages, so even when the end result is empty, one of the stages in between could contain a large amount of items that is getting sorted further down the line. However it seems that the biggest bottleneck is when TC receives a large amount of items. (I read somewhere that TC pulls things such as file size and attributes since it doesn't get that from the Everything API... is that correct?)

I have also added a WINCMD.ini entry:

Code: Select all

[Configuration]
Everything=c:\tools\everything\Everything.exe

That will cause Everything to be started if not running (when Everything is enabled for a search).
But that is not relevant for the current issue...

Yea I have such an entry as well, one of the steps I made before posting here

If you try the latest 9.50 beta release, it will show you in the statusbar which command is send to Everything.
(see this thread)
Maybe that will give you some extra information.

I saw that there seemed to be some new features related to Everything in the 9.50 beta, but also that there were a warning against using it with sensitive data. Not that I really got anything I couldn't afford losing, but I would rather not take unnecessary risks ya know?

BTW: I like your Everything colour scheme

Thanks
Just for info, it's (140, 0, 0) for the foreground and (0, 0, 0) for the background. Everything did the grey in the Name column itself.


Extra little note:
It's getting to be late around these parts, so I will be off for the time being.
I will check back here tomorrow once I get the time.

Just want to take a little moment to thank you both 2NotNull and 2petermad

[NotNull]

I'm not sure exactly what to look for in those...

Ah ... misunderstanding! I guess I wasn't clear enough (it was also getting rather late in my part of the world )
The idea was that you entered your original search query (*) in Total Commander and read from Everything's debug log what has been sent from TC to Everything (TC has to make changes to it to make it fitting for Everything's search syntax)

What stood out in your logs - and what you noticed already too - is that TC didn't close the IPC connection and did not start a fresh one. I think that is why older results are "leaking through".
(Similar to the idea of not hanging up the phone and dialing the same number ..)





(*)
Search for = ed: ext:mkv
Search in = c:\;d:\e:\;\\server\share1;\\server\share2

[tuska]

Does it make a difference if you use such a search query?

Code: Select all

Search for: ev:<C:|D:|E:|\\192.168.xxx.xxx\share_01\|\\192.168.xxx.xxx\share_02\> ext:mkv
Search in : blank

---

Windows 10 Pro (x64) Version 1909 (OS build 18363.535)
TC 9.50ß10 x64/x86 | Everything - Version 1.4.1.960 (x64)
☑ 'Everything' | Search queries: TC <=> 'Everything'

[tuska]

When searching in TC (Alt+F7/The "Find Files" Window) I need to check the 'Everything' box.

16.10.19 Release Total Commander 9.50 beta 1
16.07.19 Added: Find files: Automatically check 'Everything' option when using prefix ev: or ed: in search field (32/64)

... I have tried using the video filter in TC ...
... (Also of note, the "video:" macro still doesn't work in TC, not even with just the local drives) ...
... are the custom filters not supported by the integration?

A filter in 'Everything' cannot be used 1:1 for a search query in TC.

('Everything': Search > Organize Filters... > Video > Edit > Search: ext:3g2;3gp;...)
The following search query is required in TC, e.g. (--> You can save this search in TC...)

Code: Select all

ev:ext:3g2;3gp;3gp2;3gpp;amr;amv;asf;avi;bdmv;bik;d2v;divx;drc;dsa;dsm;dss;dsv;evo;f4v;flc;fli;flic;flv;hdmov;ifo;ivf;m1v;m2p;m2t;m2ts;m2v;m4b;m4p;m4v;mkv;mp2v;mp4;mp4v;mpe;mpeg;mpg;mpls;mpv2;mpv4;mov;mts;ogm;ogv;pss;pva;qt;ram;ratdvd;rm;rmm;rmvb;roq;rpm;smil;smk;swf;tp;tpr;ts;vob;vp6;webm;wm;wmp;wmv

I saw that there seemed to be some new features related to Everything in the 9.50 beta,
but also that there were a warning against using it with sensitive data.
Not that I really got anything I couldn't afford losing, but I would rather not take unnecessary risks ya know?

The pre-release is now very stable and contains fewer errors than version 9.22a.
(Die Vorabversion ist mittlerweile bereits sehr stabil und enthält sogar weniger Fehler als die Version 9.22a.)

I saw the following entry in a link from you (pastebin.com):

Code: Select all

cmdline D:\Programs\Everything\Everything.exe -startup

therefore I assume that you have done your 'Everything' installation on drive D:

On this point I can say the following:
If someone uses the 'Everything Service', then 'Everything' should be installed on the system drive(!) - this is usually C:.
Otherwise there may be problems with the 'Everything Service' after start of the PC.
I.e. the 'Everything Service' can sometimes be terminated and in such a case there is a problem with the search queries.
See: https://www.voidtools.com/forum/viewtopic.php?p=25101#p25101

Did you check on the "Services" tab in Task-Manager whether the 'Everything Service' was running when there were problems in TC?
In such a case was 'Everything' checked in TC?

[Son_Of_Diablo]

Ah ... misunderstanding! I guess I wasn't clear enough (it was also getting rather late in my part of the world )
The idea was that you entered your original search query (*) in Total Commander and read from Everything's debug log what has been sent from TC to Everything (TC has to make changes to it to make it fitting for Everything's search syntax)

...


(*)
Search for = ed: ext:mkv
Search in = c:\;d:\e:\;\\server\share1;\\server\share2

Yea I did that while I was trying different things (after the two first commands I posted) it yielded the same kind of results, looked like the command was split up into one action per location (c, d, e, server\share1, server\share2) and then matched with the search term (ext:mkv).

I can try running the command again though if you think it will be useful?

Does it make a difference if you use such a search query?

Code: Select all

Search for: ev:<C:|D:|E:|\\192.168.xxx.xxx\share_01\|\\192.168.xxx.xxx\share_02\> ext:mkv
Search in : blank

No that makes no difference in the result (However it does look like TC is sending the directory from which I started the search as an additional search term).

When searching in TC (Alt+F7/The "Find Files" Window) I need to check the 'Everything' box.

16.10.19 Release Total Commander 9.50 beta 1
16.07.19 Added: Find files: Automatically check 'Everything' option when using prefix ev: or ed: in search field (32/64)

... I have tried using the video filter in TC ...
... (Also of note, the "video:" macro still doesn't work in TC, not even with just the local drives) ...
... are the custom filters not supported by the integration?

A filter in 'Everything' cannot be used 1:1 for a search query in TC.

('Everything': Search > Organize Filters... > Video > Edit > Search: ext:3g2;3gp;...)
The following search query is required in TC, e.g. (--> You can save this search in TC...)

Code: Select all

ev:ext:3g2;3gp;3gp2;3gpp;amr;amv;asf;avi;bdmv;bik;d2v;divx;drc;dsa;dsm;dss;dsv;evo;f4v;flc;fli;flic;flv;hdmov;ifo;ivf;m1v;m2p;m2t;m2ts;m2v;m4b;m4p;m4v;mkv;mp2v;mp4;mp4v;mpe;mpeg;mpg;mpls;mpv2;mpv4;mov;mts;ogm;ogv;pss;pva;qt;ram;ratdvd;rm;rmm;rmvb;roq;rpm;smil;smk;swf;tp;tpr;ts;vob;vp6;webm;wm;wmp;wmv

I did no know this. I have to admit that I haven't gone through the change log, I mostly looked at the forum and the wiki.. That is on my part.
As for the filters/macros, I thought that might be the case, but since the build-in filters worked (such as dm for date modified... Or at least seemed to work) I was hoping the same would be the case for custom macros.
Thank you for the info thought

I saw that there seemed to be some new features related to Everything in the 9.50 beta,
but also that there were a warning against using it with sensitive data.
Not that I really got anything I couldn't afford losing, but I would rather not take unnecessary risks ya know?

The pre-release is now very stable and contains fewer errors than version 9.22a.
(Die Vorabversion ist mittlerweile bereits sehr stabil und enthält sogar weniger Fehler als die Version 9.22a.)

Well, I might go ahead and update then, maybe that can improve my situation.
I was deterred by the warnings on the download page, however now I'm thinking they are more for the dev(s) to cover their bases in case something unforeseen were to happen?

WARNING: This is a pre-release version. Do not use this version with critical data!
Note: For critical data, please use Total Commander 9.22a final!

As you can understand, these messages can be worrisome when you don't feel like you actually need to update.

I saw the following entry in a link from you (pastebin.com):

Code: Select all

cmdline D:\Programs\Everything\Everything.exe -startup

therefore I assume that you have done your 'Everything' installation on drive D:

On this point I can say the following:
If someone uses the 'Everything Service', then 'Everything' should be installed on the system drive(!) - this is usually C:.
Otherwise there may be problems with the 'Everything Service' after start of the PC.
I.e. the 'Everything Service' can sometimes be terminated and in such a case there is a problem with the search queries.
See: https://www.voidtools.com/forum/viewtopic.php?p=25101#p25101

Did you check on the "Services" tab in Task-Manager whether the 'Everything Service' was running when there were problems in TC?
In such a case was 'Everything' checked in TC?

Yes, I have made sure that the service is running and was running at the time.
I could try reinstalling Everything on my C drive if you believe that could have any impact on the issues.

Also yes I made sure that 'Everything' was checked in TC every time I used the search function.
A few times I also unchecked and rechecked the option just in case the flag wasn't set, and the UI was showing me something wrong.