[FYI] Eliminating the delay when deleting in-use files/dirs

[icfu]

Edit: This problem is solved by patching shell32.dll. Scroll down to http://www.ghisler.ch/board/viewtopic.php?p=88139#88139

-------------------

Well, don't get too optimistic as this is just an interesting article including an example driver that seems to work for a specific Windows version only. When trying to install it I get an "Error calling CreateService: 1073" error, German SP2. Maybe you have more luck with your version?

Anyway, maybe there is someone among you developer brainies that can explain the dumber ones among us if there is a realistic chance that we will be able to get rid of that annoying 10 seconds delay in Total Commander one day, which is STILL exactly FIVE SECONDS MORE than in good old Explorer.

If *at least* ghisler would eliminate that annoying overhead...

http://www.codeproject.com/system/NoDeleteDelay.asp

Icfu

[Lefteous]

2icfu

An interesting article.

Beside that you can be sure that the betatesters will do their best to suggest this to Ghisler.
If they fail this must be bumped during the public beta test.


BTW: My idea is the following: Ask the user what to do when the first attempt to delete the files has failed.

[majkinetor !]

I am working on that article for some time now while I was trying to create Unlocker for TC, after my conversation with Alex

It turned out that this can only be done in kernel mode, that is , driver.
I created console application that produce the same output as handle.exe, but it had do to with undocumented Native API functions and the one which is used to create snapshot of all open files in the system is hanging the system when called from user mode. Those interested can search for my posts on Sysinternals forum.


There is freeware, opensource application called
WhoUses.exe with c++ MFC classes, NTSystemInfo that does handle.exe output in user mode. However, it is very slow, otput can usualy take from 7 to 14 seconds so it can't be used as content plugin even with caching. I rewriten application to achieve speed, but as I said, nothing can fix the speed in user mode, only in kernel mode. My console test application did it for .5 seconds, but on slow computers it doesn't list all files and it can take from 10 to 16 seconds.... I was very disappointed by this result, so I started to learn about drivers. Now I have problem to get WDK. If anybody can help me retreive one, I am very familiar with things that must be done to integrate unlocking with TC, after almost 2 months of research.

The article that icfu pointed out is excellent and if one wants to create application that will handle this delay it also need to be done in kernel mode... driver again... I also suggest readers to visit related Mark Rusinovich blog which is good tutorial how to troubleshout such problems

If anybody wants to test console lock info that I did up to now this is the link (doesn't list binary files (so called modules) since that is an easier part, and they are treated with different procedures)

http://rapidshare.de/files/20839081/ConUnlocker2.exe.html (~50K)

Function that will try to get filenames of open handles is problematic so I execute it in a thread and kill a thread after 100ms if it doesn't return. That is the reason for smaller number of listed files then in handle.exe which runs in kernel where this function doesn't hang (I guess nobody at MS wanted to use this function from user mode, and I guess that such thing wasn't even planned while implementing mentioned function). I already implemented everything that is need to produce output for content plugin and to create dll that will do acctual unlocking of locked files. So, again, if anybody have WDK I will try to translate everything to driver.


This is part of the output on my system:

NO:446 | 23 H:1904 PID:4
P : System
FN: C:\Documents and Settings\LocalService\Local Settings\Application Data\Microsoft\Windows\UsrClass.dat
----------------------------------
NO:890 | 24 H:1104 PID:600
P : C:\WINDOWS\system32\csrss.exe
FN: C:\WINDOWS\system32\ega.cpi
----------------------------------
NO:1917 | 25 H:2624 PID:624
P : C:\WINDOWS\system32\winlogon.exe
FN: C:\WINDOWS\inf\syssetup.PNF
----------------------------------
NO:2140 | 26 H:688 PID:668
P : C:\WINDOWS\system32\services.exe
FN: C:\WINDOWS\system32\config\AppEvent.Evt
----------------------------------
NO:2144 | 27 H:704 PID:668
P : C:\WINDOWS\system32\services.exe
FN: C:\WINDOWS\system32\config\SecEvent.Evt
----------------------------------
NO:5797 | 48 H:724 PID:368
P : C:\PROGRA~1\MI6841~1\MSSQL\Binn\sqlservr.exe
FN: C:\Program Files\Microsoft SQL Server\MSSQL\Data\tempdb.mdf
----------------------------------
NO:6012 | 49 H:372 PID:940
P : C:\Program Files\Eset\nod32krn.exe
FN: C:\Program Files\Eset\logs\virlog.dat
----------------------------------
NO:6013 | 50 H:376 PID:940
P : C:\Program Files\Eset\nod32krn.exe
FN: C:\Program Files\Eset\logs\warnlog.dat
----------------------------------
NO:6034 | 51 H:460 PID:940
P : C:\Program Files\Eset\nod32krn.exe
FN: C:\Program Files\Eset\cache\CACHE.NDB
----------------------------------
NO:2511 | 29 H:436 PID:680
P : C:\WINDOWS\system32\lsass.exe
FN: C:\WINDOWS\Debug\PASSWD.LOG
.....

[majkinetor !]

BTW: My idea is the following: Ask the user what to do when the first attempt to delete the files has failed.

My idea:

1. realize that Ghisler will probably not do that... (please surprise me here)
2. ... which means that somebody else has to do it.
3. The acctual idea:
- Install system hook that will monitor TC delete dialog
- when recognised, grab the files (copy selection to clipbard, take file names from clipboard) and test for locks.
- if locks are found, display applications that hold the lock, and unlocker dilalog for selection. Kill TC's delete dialog.
- make this behavior configurable (option to turn lock check off, for those who for some lunatic reason doesn't want that)


Ghisler should at least do the following.

1. Before delete isue CrateFile with read only attributes.
2. If GetLastError() say that file is locked, skip this file, and leave it selected. Eventually call custom application for selection (so, for instance, little program ForceDel can be called on selection)

[XPEHOPE3KA]

copy selection to clipbard, take file names from clipboard

Not a perfect solution - consider problems which OpenFile4TC has now: it has to remember user's clipboard contents no matter whether it is text or image etc., then restore it and see if localized text remains localized (not becomes abracadabra).
But I must admit these problems are avoidable.

Anyhow, Ghisler uses his own deleting algorithm, so it must be improved by himself: at least a user should have an option to select how many times Ghisler's analog of _IsDirectoryDeleteable procedure is invoked (I doubt that he uses windows' internal function - otherwise the delay would have been the same).

[majkinetor !]

Anyhow, Ghisler uses his own deleting algorithm

so ?
this part comes before delete not in or after...

Not a perfect solution

of course.... that's why we wanted scripting in TC in first place. Because we want to avoid things like this.

Perfect solution is so simple that it hurts: when TC recieves WM_USER+X ::= WM_GETCURRENTSELECTION, it sends current selection back to caller, via WM_COPYDATA message.

Now everybody will jump and screem.. .WHAAATTT , viruses, trojans, polar bears, eleph... (we will skip them this time) are going to get my selection and DELETE IT FOREVER !!!!! Woaaaaaa.. what an agly trip.

But trip it is!
Viruses, trojans, rootkits, are today so sofisticated that there is nothing you can do protect yourself. Grabing current selection from TC or using eventual TC's internal script to kill someones day will NOT become security hole, since you can do all kind of malitious things to TC users today, without scripting or any kind of support for 3th party tool communication with such easy that it can do anybody who understand what Notepade is. So those are lousy excuses. And ofcourse, those should be OFF by default, and one big screen saying ARE YOU NUTS ? should popup when you click checkbox - that should eliminate any missunderstanding...

[icfu]

I have successfully patched the shell32.dll on my system and enjoy the following delays for deleting in-use directories AND files now:

Explorer: 0 seconds!
TC: 0 seconds! (with both Explorer and TC deletion method)

Yeah, that rocks!

What to do:
You are responsible for your own actions so do lots of system backups before trying that or make sure you know what to do in case anything goes wrong!

1. Make sure that only one shell32.dll exists on your system in \windows\system32\. Rename or delete the one possibly located in \windows\system32\dllcache and remove XP CD in case it is inserted.
2. Open a copy of \windows\system32\shell32.dll with a hex editor of your choice, for example Tiny Hexer (free)
3. Search for the hex string 8bff558bec565733ffeb2183ff05, you will find it twice! In each occurence replace the last 05 with 00. Make sure that you overwrite and not insert!
4. Save the changes!
5. Rename \windows\system32\shell32.dll to, for example, shell32.dll.bak. Renaming of in-use files is no problem in XP! This info is unfortunately missing in the above article...
6. Copy the formerly saved shell32.dll in \windows\system32\
7. Reboot and have fun!

What to do next:
Do an autopatcher that automates these tasks. In case anyone knows a decent one (simple, tiny), please drop me a line.

Icfu

[majkinetor !]

what about windows file protection ?
I guess it must be also off

[XPEHOPE3KA]

2m
No it mustn't. Strangely, but when I was replacing notepad with akelpad, when I deleted the former from dllcache directory - it immediately reappeared! But when I deleted shell32.dll I didn't reappear.

2icfu
Now that rocks me too! Thank you greatly!

[icfu]

I have edited my posting, also works for files! Yeehaw!

Icfu

[Lefteous]

2icfu
What I don't understand is why the delay in TC is reduced to 0 seconds. I thought TC tries to call DeleteFile (the 2nd attempt of the two) which is located in the kernel
Well maybe "deleting the file directly" doesn't mean DeleteFile but SHFileOperation without FOF_ALLOWUNDO is called to delete the files.

[Alextp]

majkinetor

If anybody wants to test console lock info that I did up to now this is the link (doesn't list binary files (so called modules) since that is an easier part, and they are treated with different procedures)
...

Good work...

I'll try to test it. When you finished with it and are planning to make content plugin (finally), I may suggest to make it as part of LockedTest plugin (it's in Delphi. Drop me a line in email, I'll send you source). So next version of LockedTest will be yours.

[majkinetor !]

Hej Alex.

Good work...

We will se about that.... nothing is finished yet. At this stage, this can be used for faster computers (>= 2.5GHz, todays standard, if I may say, even in Belgrade). If I cache data, that is, if I create snaphsot of all open files on first run and retreive this info for subsequent files, this can be fast - it will produce about 2 sec delay on first screenshot, and less on subsequent. Cache time, should easily be about 1 minute, so you can use same lock info on fast directory changes without any performance loss.

As I am purist, this stage doesn't satisfy me. Driver is only good solution and since I don't know anything about drivers, this is good oportunity to learn something (I already started). I hope that I will not lose interest in this, since it already took me too much time, and when I realised that it will finish in kernel, I was ready to give up (like Sanskritfritz already sugested me, too much work, too little results... right).

BUT, I must say that knowing how system function internally, created new dimension in my Windows usage. I recommend you to do the same, if you want to go to the next level of Windows knowledge.... some really interesting stuff are there.... and you are limitless... in kernel, there is no security.... So all that gave me inspiration to continue....

I will send you the source if you are interested (in c).

Also I am not interested only in content plugin. If Ghisler doesn't want to cooperate with our efforts, I am planing to change TC's behavior on delete - lock info is nothing, if you can't use that information to unlock files... If everything here finish OK, I am willing to create dll that Ghisler can use to provide infomation about locks in the system when encountering files that can not be deleted. He just need to say some positive words about that.

Please, tell me results of your test when you do it. Be aware that modules will not be listed (exe, dll, ocx ...) There are regular PSAPI functions that can retreive this, and there are faster undocumented but this is generaly no problem.

2 X
That behavior is strange. One possible explanation is that shell components are maybe special case because they are part of the SHELL which is replacable by arhitecture, so if file protection was on for those, all 3thd party shells would have problems to replace those files... I guess that can be the case here...

[SQUIRE]

I enjoyed reading Dan Farino's article - it's an interesting walkthrough of the debug process.

What puzzles and worries me a little is that no one seems to have asked why there is a delay loop in the first place. No sane programmer would deliberately insert such a long wait just for fun, especially when the consequences are so clearly apparent to a user. Why go to all the trouble of writing and debugging all that extra code? And why specifically 5 secs, not 1 or a few milliseconds? It cannot have passed beneath MS radar as it's so obvious, and it would be child's play to remove if it wasn't needed. So why has it been allowed to remain for so long?

I searched MS Knowledge Base for a clue and it's strangely silent on the issue. One of the responses to the article alluded to a possible race condition (maybe uncommon) being a probable reason. It also occurred to me that such a delay might be a legacy of working with older and slower file storage media like floppies and tape. It might be there to allow for spin-up time for example.

That delete function probably sees extremely heavy use, not only by a user sitting impatiently at his KB but also by unattended applications performing deletes and moves. Cleaning up of installer temp files springs to mind. I wonder if there's a possibility of unwanted files remaining undeleted if the function is too quick to say 'No' ?

Maybe Ghisler has studied the consequences of too rapid a response in depth and can give a definitive answer. TC would appear to hit the function twice, hence the 10 second delay.

[icfu]

Why go to all the trouble of writing and debugging all that extra code?

Because they had thought that if people wanted to delete a file, they would prefer to have a timeout, a chance to unlock the file manually, over an error message that comes too quickly. Maybe it's also a psychological thing like: "That was too fast, you haven't tried your best to nuke that file. Hey, f*uckin' Windows, try again before annoying me with popups!"

And why specifically 5 secs, not 1 or a few milliseconds?

Bill: 0 is too few, 10 is too much, let's take the middle.
Steve: All right, you are the boss.

Think about all the hype if 23 tries would be made...

Cleaning up of installer temp files springs to mind. I wonder if there's a possibility of unwanted files remaining undeleted if the function is too quick to say 'No' ?

What do installers do? If something can not be deleted, the user is told to reboot (annoying and usually superfluous under XP because every action that will be done after reboot can be extracted from registry and done manually as well) or to manually remove the remaining files.
The only thing that changes from the application's point of view is that the answer from the OS, if a file is locked or not, comes faster. Some installers suggest a retry or retry themselves when they get negative feedback, why should that be a problem?

Anyway, regard me as your testing dummy. If any problems arise, you will hear from me.

Maybe Ghisler has studied the consequences of too rapid a response in depth and can give a definitive answer. TC would appear to hit the function twice, hence the 10 second delay.

ghisler has already given an answer in this REQ thread from me (German):
http://www.ghisler.ch/board/viewtopic.php?p=34816#34816

TC tries two times:
1. delete via recycle bin: SHFileOperation()
2. delete directly: DeleteFile() (this is what is invoked when using Shift-DEL)

2 is needed because some drives don't support recycle bin (floppy for example).

This is a severe problem imho, unfortunately ghisler hasn't commented my concern in that thread:
If you try to delete a locked file to the recycle bin (from a supported drive) and successfully manage to unlock the file when DeleteFile() is executed (between 5 and 10 seconds waiting time), your file will not wait for you in the recycle bin but will be nuked by DeleteFile()...

Icfu