Speedup File Search!

[Shark321]

Hi!

In the latest issue of the German c't magazine they show a freeware file search tool called Everything (google for voidtools).

The tool does file search by looking only on the MFT on NTFS partitions. This is what I've been looking for, for a long time, so I've tried it. It really does the job, it needs 1 second to do a complete search on my 2 TB harddrive (with a lot of files). Total Commander needs more than 60 seconds for the same job.

The immediate thought was - why does Total Commander (which I use every day, regged user) does not the file search, and even directory listing in the same manner, by looking only at the master file table and nothing else? From a coding standpoint it should not be a problem and would give a huge speedup!

[Lefteous]

The question is: Do we want an elevated scheduled task that does the dirty work or not?

[MVV]

Yes, Lefteous is right, MFT may only be accessed from process started with elevated rights.
But AFAIK Everything creates some kind of database when scanning MFT. And this database may be accessed w/o elevated rights (if it isn't encrypted). Also Evething's database may be exported as text file.
BTW, here you may find plugin that uses Everything for searching. But it also requires elevated rights...


Also it is possible to setup Everything to open selected file/folder in TC. Edit Everything.ini (close service before):

Code: Select all

open_file_command=$exec("C:\TOTALCMD\TOTALCMD.exe" /O /S /L="%1")
open_folder_command=$exec("C:\TOTALCMD\TOTALCMD.exe" /O /S /L="%1")

After theese changes Everything on double-click file/folder will make TC jump to theese file/folder (open it in active tab or start new TC if not started - but new TC will be started elevated because Everything is elevated). You may add parameter /T to open each search result in new tab, also you may do this with another commands - e.g. open you make like this but Explore you make using new tab:

Code: Select all

explore_folder_command=$exec("C:\TOTALCMD\TOTALCMD.exe" /O /T /S /L="%1")
explore_folder_path_command=$exec("C:\TOTALCMD\TOTALCMD.exe" /O /T /S /L="%1")

At least, (but better case for TC users)
Idea to realize:
For QS TC already supports external search function in DLL, so I think for file search something similar may be realized.

Technical suggestion:
TC loads some DLL like for QS and calls its function when user searches files. Exported functions: FindFirstFile(filemask, search_paths, var data); FindNextFile(handle, var data) and FindClose(handle). Plugin returns one filepath on each call, and TC processes results and filters it using advanced search options and plugins). Returned information for each file may contain just file path or some structure with more information.

Of course, for Vista/7 plugin should work w/o elevation (but here I think some admin tool is needed because e.g. Everything needs elevation).

Why we need it?
This will be a good way to use quick-search engines like Everything wthn internal search feature. This will be fast and powerful.

[Samuel]

2MVV:
Support++

[m^2]

Support++ for MFT searching. Though I believe it was suggested already. Indexing was for sure.

[tosbsas]

+++++

[m^2]

I just found that there's a tool that does it on sourceforge.
It's GPL, so it can't be used in TC directly (unless TC goes open source ), but it's still a cheap know-how.
Or TC could create an interface for replacing the default search engine (like with quick search now) and then sb. (i.e. Christian Ghisler) could create an external tool to perform this task.

[MVV]

Or TC could create an interface for replacing the default search engine (like with quick search now) and then sb. (i.e. Christian Ghisler) could create an external tool to perform this task.

I suggested this way because it is an easiest way to get MFT searching feature in TC, also using this way each may use its favorite MFT searching engine.
But please note that such DLL should work w/o elevation to allow using un-eleveated TC. So since MFT access requires elevation, DLL should have some link with elevated tool that will do MFT searching or access to external MFT search engine like Everything - for Everything there is command-line search tool with sources available, but it also requires elevation.

[Boofo]

This is what I have been using in my Everything.ini file for quite a while now and it works great.

Code: Select all

open_folder_path_command=$exec("%COMMANDER_PATH%\TOTALCMD.EXE" /O /T /L="%1")
open_file_command=$exec("%COMMANDER_PATH%\TOTALCMD.EXE" /O /T /L="%1")
open_folder_command=$exec("%COMMANDER_PATH%\TOTALCMD.EXE" /O /T /L="%1")

You have to be using the latest alpha for the COMMANDER_PATH parameter to be used, though.

[MVV]

$exec("%COMMANDER_PATH%\TOTALCMD.EXE" /O /T /L="%1")

This will always jump to folder in left TC panel. With additional /S parameter TC will open folder in active panel.

You have to be using the latest alpha for the COMMANDER_PATH parameter to be used, though.

AFAIK, if Everything wasn't started from TC or its child process, it will know nothing about COMMANDER_PATH. Or latest version of Everything may detect TC installation (that's strange thing even if so)? But what if you have more than one TC installed?

[Boofo]

2MVV,

Yes, I know about the /S parameter. I like it opening up in the same pane all the time.

I stand corrected on the path variable. I forgot that I added the path to my Windows path statement. I made a reg file to do it automatically.

Code: Select all

Windows Registry Editor Version 5.00

;This will set the Total Commander path in the system variables.

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Environment]
"COMMANDER_PATH"="C:\\Program Files\\totalcmd"

[m^2]

Sadly, external tools don't cut it. They don't have all the features, even the simplest one like putting all results in the TC panel.

[MVV]

Sadly, external tools don't cut it. They don't have all the features, even the simplest one like putting all results in the TC panel.

This will work in case of external search DLL - TC will call DLL function to retrieve next search ressult and then add to listbox if need.

[Lefteous]

2MVV
Plugin interfaces make sense when there are almost unlimited ways of performing a task. Content, packer, lister and file system plugins are good examples for such almost unlimited possibilties.

In this case I don't see why we need such an interface. TC should know how to search with good performance out of the box.



In addition there are much more things to improve in search as just the performance (e.g.):
- Search UI
- Persistence
- Concurrency
- Result handling UI

I hope an improved search will be TC 8 top feature.

[MVV]

In this case I don't see why we need such an interface. TC should know how to search with good performance out of the box.



In addition there are much more things to improve in search as just the performance (e.g.):
- Search UI
- Persistence
- Concurrency
- Result handling UI

I hope an improved search will be TC 8 top feature.

But TC has DLL interface for quick search feature - do you think that quick search filtering is more important than quick file search?