full Win 64bit support to come?

[levicki]

Yes, it doesn't say specifically, but it says that you have to reenable the redirection as soon as you have the file handle.

Have you by any chance read this?

This function (Wow64EnableWow64FsRedirection, i.e. the one you pointed to) may not work reliably when there are nested calls. Therefore, this function has been replaced by the Wow64DisableWow64FsRedirection and Wow64RevertWow64FsRedirection functions.

Could that be the reason why you had trouble with it?

Anyway, I can assure you that some things really don't work correctly if you leave the redirection on - I've coded a few things about it a few months ago. I don't remember exactly the situation, but there's a reason.

I am sorry but reason without a proof doesn't stand a chance in my book. It is called a myth.

[gigaman]

Yes, I've read it - but if you only disable the redirection, there's hardly any nesting. So no, I don't believe it's got anything to do about it. Note that nothing has changed in the example for the new functions - the redirection is re-enabled right after CreateFile() call.

I'll do some checking when I have a while.

Btw, how exactly were you loading the library?

[levicki]

Yes, I've read it - but if you only disable the redirection, there's hardly any nesting. So no, I don't believe it's got anything to do about it. Note that nothing has changed in the example for the new functions - the redirection is re-enabled right after CreateFile() call.

I'll do some checking when I have a while.

Btw, how exactly were you loading the library?

Dynamically via LoadLibrary() of course. That is how TC (or any other program for that matter) loads plug-ins if I remember correctly.

Remember that the redirection is disabled only for the thread which calls Disable() so if you had two threads (i.e. worker and GUI) you might have ended up with mixed redirection state.

[gigaman]

Dynamically via LoadLibrary() of course. That is how TC (or any other program for that matter) loads plug-ins if I remember correctly.

Well, sure - I rather meant what library was that, if you specified the full path, etc. And yes, TC uses LoadLibrary() to load plugins - which is actually wrong in my opinion; I'd find LoadLibraryEx(.., .., LOAD_WITH_ALTERED_SEARCH_PATH) better, because you wouldn't have to install 3rd party libraries, needed for TC plugins, directly into TC directory.

Anyway, one of the problem is that GetSystemDirectory() always returns "Windows\System32" - no matter if the redirection is disabled or enabled. So, if you want to dynamically load a system library using a full path (and not just the filename - so you call GetSystemDirectory() and append the filename), it will fail when the redirection is disabled. Since you can't control other modules (e.g. plugins) you don't have the sources for, you can't say this won't happen.

[ghisler(Author)]

What if a plugin uses dlls from system32 which Total Commander does NOT use? Will the LoadLibrary fail then when the path has been redirected?

[gigaman]

I'm afraid I don't understand the question.

When the redirection is enabled (which is the default), all access to Windows\System32 folder goes to Windows\SysWOW64 instead - the folder which holds the 32bit versions of the DLLs. So, I was trying to say that when the redirection is disabled (so the subsequent access goes to the real Windows\System32 folder with the 64bit versions of the DLLs) and the executed code (not aware of Win64) uses this path to load a DLLs, it will actually try to load a 64bit library - which is not possible.

[levicki]

Well, sure - I rather meant what library was that, if you specified the full path, etc.

I tried with my custom inject.dll which was in the same folder as the .exe, and I tried with msvcrt.dll which was in 32-bit system32 folder. Both loaded without errors (i.e. I got valid module handle). I specified just the filename.ext as you should always do, because hard-coding any paths is evil.

And yes, TC uses LoadLibrary() to load plugins - which is actually wrong in my opinion; I'd find LoadLibraryEx(.., .., LOAD_WITH_ALTERED_SEARCH_PATH) better, because you wouldn't have to install 3rd party libraries, needed for TC plugins, directly into TC directory.

You don't have to put them into TC directory anyway and there is really no need to use altered search path. You can put those DLLs in system folder and they will be found.

When the redirection is enabled (which is the default), all access to Windows\System32 folder goes to Windows\SysWOW64 instead - the folder which holds the 32bit versions of the DLLs.

You are mixing things, API name is Wow64DisableWow64FsRedirection(). Fs stands for filesystem. So just the filesystem API can be affected by this -- CreateFile(), whatever works with file HANDLEs.

So, I was trying to say that when the redirection is disabled (so the subsequent access goes to the real Windows\System32 folder with the 64bit versions of the DLLs) and the executed code (not aware of Win64) uses this path to load a DLLs, it will actually try to load a 64bit library - which is not possible.

gigaman, I believe you don't have correct info on this.

First, LoadLibrary() is exported from KERNEL32.DLL. When you start a 32-bit process, it gets \WINDOWS\SYSWOW64\KERNEL32.DLL mapped in its address space regardless of the redirection state because real 64-bit KERNEL32.DLL could not be mapped anyway -- it requires 64-bit types as arguments.

Second, once the KERNEL32.DLL is mapped into process address space, it is not unmapped until the process exits.

From those two points one simple logical consequence can be drawn -- when you call LoadLibrary() you are always calling 32-bit version which will surely try (and succeed in) loading of 32-bit DLLs.

GetSystemDirectory() is obsolete and should not be used anymore if at all possible. LoadLibrary() searches system directory even without you adding it to the file path.

If in doubt, you should check Dynamic-Link Library Search Order.

If for some reason LoadLibrary() can't find a DLL, you can explicitly specify additional search path using SetDllDirectory() and from that point on you will have alternate search order which is listed here under Remarks section.

Anyway, why are we wasting time on discussion when Christian could compile new TC version just with Wow64DisableWow64FsRedirection() call at the start and email it to me for testing? You can tell me which plugins you suspect to fail and I will test them and report here so he will know whether he can safely implement my suggestion or not. Of course, if he wants to. I am available with my XP x64 for testing.

[gigaman]

And yes, TC uses LoadLibrary() to load plugins - which is actually wrong in my opinion; I'd find LoadLibraryEx(.., .., LOAD_WITH_ALTERED_SEARCH_PATH) better, because you wouldn't have to install 3rd party libraries, needed for TC plugins, directly into TC directory.

You don't have to put them into TC directory anyway and there is really no need to use altered search path. You can put those DLLs in system folder and they will be found.

Sure, but that's even worse in my opinion - why should I put plugin-specific DLLs into system folder?

When the redirection is enabled (which is the default), all access to Windows\System32 folder goes to Windows\SysWOW64 instead - the folder which holds the 32bit versions of the DLLs.

You are mixing things, API name is Wow64DisableWow64FsRedirection(). Fs stands for filesystem. So just the filesystem API can be affected by this -- CreateFile(), whatever works with file HANDLEs.

I know that, and that's exactly what I mean. But of course, there are lots of functions working with file names (or you have to open the file first) - that's why I wrote "all access".

So, I was trying to say that when the redirection is disabled (so the subsequent access goes to the real Windows\System32 folder with the 64bit versions of the DLLs) and the executed code (not aware of Win64) uses this path to load a DLLs, it will actually try to load a 64bit library - which is not possible.

gigaman, I believe you don't have correct info on this.

I'd say you just didn't get my point
What I meant is e.g. the following snippet (as simplistic as it can be):

Code: Select all

CHAR szPath[MAX_PATH];
GetSystemDirectory(szPath, MAX_PATH);
strcat(szPath, "\\MyDll.dll");
LoadLibrary(szPath);

Now, this code will always load X:\Windows\System32\MyDll.dll. Now, if this code is placed in a plugin (which is not aware of Win64, i.e. the DLL it really wants is actually in SysWOW64) and you call this code with the filesystem redirection disabled (e.g. if you disable the redirection and then call LoadLibrary(the_plugin_dll), it will fail - because it will be accessing the 64bit system folder.
Yes, you can fix that, e.g. by avoiding the GetSystemDirectory() call, but as I said - suppose it's in a plugin you don't control (besides, I don't think GetSystemDirectory is obsolete - loading the library this way might be better security-wise, because you don't risk somebody putting a forged DLL into the folder of your application - but that's not the point here).

[levicki]

I am asking all of you what is more important here -- proper support for a new operating system or backward compatibility with poorly written plugin(s)?

Lets face the simple facts again:

1. You can't change the way the OS works
2. You can influence plugin writers and even change the API
3. Many of them publish the source code so it can be fixed

So will you go fight the windmills or you will do what has to be done if you expect people to continue using TC?

Sure, but that's even worse in my opinion - why should I put plugin-specific DLLs into system folder?

Because that is the place system will search for them regardless of the path you specify?

I'd say you just didn't get my point
What I meant is e.g. the following snippet (as simplistic as it can be):

Code: Select all

CHAR szPath[MAX_PATH];
GetSystemDirectory(szPath, MAX_PATH);
strcat(szPath, "\\MyDll.dll");
LoadLibrary(szPath);

Sigh, I already told you that you don't have to append path. It will look there anyway. Anyway, the above code is broken because of possible buffer overflow -- GetSystemDirectory() can return MAX_PATH characters and you will blindly append "\\MyDll.dll" to it thus trashing the stack.

Now, this code will always load X:\Windows\System32\MyDll.dll. Now, if this code is placed in a plugin (which is not aware of Win64, i.e. the DLL it really wants is actually in SysWOW64) and you call this code with the filesystem redirection disabled (e.g. if you disable the redirection and then call LoadLibrary(the_plugin_dll), it will fail - because it will be accessing the 64bit system folder.

First, when you do LoadLibrary(the_plugin_dll) only its DllMain() gets called. All references to other DLLs it needs are resolved while image is loaded and mapped into memory. Since the process is 32-bit, the plugin will get all of its DLLs loaded properly.

Second, if the_plugin_dll uses LoadLibrary() itself explicitly it should never do it from DllMain(). It is strictly forbidden to do it because you are easily caught in a race condition.

If the_plugin_dll fails in DllMain() (that is if it fails when you call LoadLibrary(the_plugin_dll)) then it is BROKEN and its failure has nothing to do with the redirection itself.

The only place it should fail because of redirection is if you call an exported function from the_plugin_dll which contains the above code snippet. I have tested it and it indeed fails. If it weren't using GetSystemDirectory() and if it kept its DLLs in the same folder it wouldn't fail.

I have asked one of the Microsoft developers for help because I think they could implement a workaround. In the meantime, Christian could implement some import table patching for the plugins. In other words, he could patch their GetSystemDirectory() to point to his helper.dll which exports that function and which always returns correct path. It would be a bit more work but it is doable. I may even provide him with a code sample.

[gigaman]

I am asking all of you what is more important here -- proper support for a new operating system or backward compatibility with poorly written plugin(s)?

Lets face the simple facts again:

1. You can't change the way the OS works
2. You can influence plugin writers and even change the API
3. Many of them publish the source code so it can be fixed

So will you go fight the windmills or you will do what has to be done if you expect people to continue using TC?

OK, I have a different question. Why did Microsoft put the system redirection there in the first place? Because they wanted to maintain the compatibility - they wanted the existing 3rd party application to keep working without updating/fixing/recompilation - no matter what hardcoded paths or nasty tricks they use. That's the point of the redirection. Now, you suggest to disable the redirection and then pass control tol 3rd party code, probably not Win64-aware. [This 3rd party code might be the plugins, other extensions, and maybe also other Microsoft DLLs or Borland packages - you can't be 100% sure the 32bit versions of these files are OK]. So, you are simply denying the Microsoft's concept... All I'm saying is that you should keep the system (thread redirection status in this case) on the default value if you're calling call you don't know/control. It is that hard to keep the rules?

Sure, but that's even worse in my opinion - why should I put plugin-specific DLLs into system folder?

Because that is the place system will search for them regardless of the path you specify?

Do you install all applications directly into Windows or System folder, without any subfolders, etc? I just prefer to keep the file/folder structure organized, that's all.

I'd say you just didn't get my point
What I meant is e.g. the following snippet (as simplistic as it can be):

Code: Select all

CHAR szPath[MAX_PATH];
GetSystemDirectory(szPath, MAX_PATH);
strcat(szPath, "\\MyDll.dll");
LoadLibrary(szPath);

Sigh, I already told you that you don't have to append path. It will look there anyway. Anyway, the above code is broken because of possible buffer overflow -- GetSystemDirectory() can return MAX_PATH characters and you will blindly append "\\MyDll.dll" to it thus trashing the stack.

Man, I said "as simplictic as it can be". Yes, there should be error checking and bounds checking there - I know that. I was trying to make it as short as possible to illustrate an idea, not to write a real application. And I also wrote that I know it would work without the prepended path - but again: suppose this is an existing 3rd party code you don't control.

Now, this code will always load X:\Windows\System32\MyDll.dll. Now, if this code is placed in a plugin (which is not aware of Win64, i.e. the DLL it really wants is actually in SysWOW64) and you call this code with the filesystem redirection disabled (e.g. if you disable the redirection and then call LoadLibrary(the_plugin_dll), it will fail - because it will be accessing the 64bit system folder.

First, when you do LoadLibrary(the_plugin_dll) only its DllMain() gets called. All references to other DLLs it needs are resolved while image is loaded and mapped into memory. Since the process is 32-bit, the plugin will get all of its DLLs loaded properly.

Second, if the_plugin_dll uses LoadLibrary() itself explicitly it should never do it from DllMain(). It is strictly forbidden to do it because you are easily caught in a race condition.

If the_plugin_dll fails in DllMain() (that is if it fails when you call LoadLibrary(the_plugin_dll)) then it is BROKEN and its failure has nothing to do with the redirection itself.

The only place it should fail because of redirection is if you call an exported function from the_plugin_dll which contains the above code snippet. I have tested it and it indeed fails. If it weren't using GetSystemDirectory() and if it kept its DLLs in the same folder it wouldn't fail.

Yes, I know that and it's actually what I meant, I was just trying to make the post shorter and didn't run into all the details (I mainly wanted to distinguish between the_plugin_dll and MyDll). But again, the whole redirection feature is there because people don't write fully correct code. So, if something worked on Win32 - why break the compatibility on Win64 when it's not necessary?

I have asked one of the Microsoft developers for help because I think they could implement a workaround. In the meantime, Christian could implement some import table patching for the plugins. In other words, he could patch their GetSystemDirectory() to point to his helper.dll which exports that function and which always returns correct path. It would be a bit more work but it is doable. I may even provide him with a code sample.

Erm, I'm obviously missing something. First, there are probably dozens of other ways a strangely written application/plugin may fail with system32 redirection disabled (if nothing else, then the hardcoded "system32" subfolder, getting the system folder path somewhere from registry, ...), second - why implement something that crazy (and incompatible - e.g. with various executable packers), instead of simply enabling the redirection again when you're done with whatever necessary in the real system32 folder?

[levicki]

Lets cut to the chase because I am getting bored.

1. Microsoft did the redirection thing but it didn't cover all the cases. When the redirection is enabled GetSystemDirectory() should obviously return SysWow64 instead.

2. Having to save and restore redirection state is inefficient, troublesome and impractial to say the least. You have to do it before and after any plugin call and you must remember to use the same thread again. Either that or use separate thread for loading and calling plugins which I doubt Cristian would ever want to do.

3. Alternatives with IAT patching exist but they are harder to implement so it is unreasonable to ask that. Additionaly it may not work with delay-loaded DLLs becase they don't have IAT at all.

4. Using TC without access to real System32 folder is not an option.

5. Using hardlinks isn't an option because many people keep their OS on FAT32 partition (lets not discuss why at the moment).

From my point of view, there are two solutions to this problem:

Righteous solution:
Microsoft accepts my suggestion and implements a hotfix for GetSystemDirectory().

Real world solution:
Christian asks every plugin developer not to use absolute paths in TC plugins as they never should've been using them in the first place.

[gigaman]

I agree with most of that (even though point 1 may actually not be an omission, but done for a reason... don't know).

I would just like to say that the cases of hardcoded "system32" folder path may be more common than we think. [For example, if you check Delphi 7 sources, you'll find out that even some components shipped with Delphi by default use that (call GetWindowsDirectory and append "system32"); OK, it's probably nothing critical in this particular case and besides, it's a subfolder of system32 where I'm not sure about the redirection status right now]. But, problems might occur even if the authors are following all the recommendations - if they use bad components/libraries.

[Lefteous]

Lefteous solution:
Go and get a fine 32 bit Windows.

[levicki]

Lefteous solution:
Go and get a fine 32 bit Windows.

Yes, and then I won't be able to develop 64-bit programs. No thanks.

[ghisler(Author)]

I agree with most of what user gigaman writes. One solution would be to allow the user to enable/disable the system32 redirection via a wincmd.ini switch. The user could then choose between using all plugins and system32 pointing to the 32-bit dlls, and using only well behaving/updated plugins and system32 pointing to the 64-bit dlls. What do you think? The current behaviour would remain the default.