[WCX] Wcx6432: 64-to-32 bit WCX plugin wrapper

[MVV]

I've understood why callbacks haven't worked, unfortunately it is not documented anywhere so it was not so easy to guess it.
I've called callbacks from new threads but it is necessary to run them from exactly that threads that have called functions. So fixing it have required serious redesigning.

Wcx6432 Beta 192

Now callbacks seem to work pretty nice.

[Horst.Epp]

I've understood why callbacks haven't worked, unfortunately it is not documented anywhere so it was not so easy to guess it.
I've called callbacks from new threads but it is necessary to run them from exactly that threads that have called functions. So fixing it have required serious redesigning.

Wcx6432 Beta 192

Now callbacks seem to work pretty nice.

Still works fine with my msi test files
Thanks for all the work.
Do you plan to also add support for WLX plugins ?
I'm interested to get the Syn2 Lister plugin in x64.

[TW]

works nicely for me too.
well done.

Do you plan to also add support for WLX plugins ?

is that even possible? the other wrappers all were wcx only too.

[MVV]

Still works fine with my msi test files

Good news.

Do you plan to also add support for WLX plugins ?
I'm interested to get the Syn2 Lister plugin in x64.

Can't you use 64-bit version of Syn2? Or it is another plugin?

works nicely for me too.
well done.

Thanks.

Do you plan to also add support for WLX plugins ?

is that even possible? the other wrappers all were wcx only too.

I think it should be possible if plugin follows WXX API and doesn't communucate with TC in other ways (e.g. by subclassing TC windows).
I think I will make wrappers for another kinds of plugins but I don't know will they work with specific plugins or not.

[deus-ex]

Hi MVV,

thank you for the latest updates of your great wrapper.

I found that during unpacking only the lower bar representing the total progress is updated, the upper bar for the per-file-progress never gets updated. I think this is by design and fine with me. Though when testing the original 32-bit plugins with Total Commander 32-bit I found, that some of the packer plugins do not draw the upper per-file-progress bar at all. Maybe that's an detail you want to add to your wrapper.

The unpacking process now runs definitely faster with the latest release v1.0.0.194 of your wrapper, compared to the previous release v1.0.0.172. Though unpacking with the wrapper takes about 7-8 times longer compared to the plugins under TC 32-bit.

For example I tested GAUP_pro.wcx unpacking the game file Hexen.wad from the game Hexen - Beyond Heretic (~19.1 MB). Using the plugin under TC 32-bit it takes ~70 seconds to unpack the entire WAD-file, under TC 64-bit with the wrapper it takes ~537 seconds. If this is related to the update frequency of the progress bar, maybe you can either improve the unpack process by reducing the progress bar update frequency, or provide an in-file with a user-configurable flag that ranges from 0 to n, where 0 = off (no progress bar update) and n = n-times per second, probably limiting it with a certain maximum value.

Also I found that when trying to access any 32-bit plugin config menu via the Pack file dialog, it stalls TC and eventually leads to a crash.

[TW]

Also I found that when trying to access any 32-bit plugin config menu via the Pack file dialog, it stalls TC and eventually leads to a crash.

ah, yes, the same here. i didn't try this.

Though unpacking with the wrapper takes about 7-8 times longer compared to the plugins under TC 32-bit.

hm... i cannot confirm this, unpacking calibre-2.58.0.msi takes about the same time on my old notebook. don't know about other plugins though, the ones i use (IMAginator etc) can handle only small files anyway.
the msi plugin isn't well optimized i have a feeling, towards the end it usually gets slower and slower... it takes some time to unpack even very small files.

I think I will make wrappers for another kinds of plugins but I don't know will they work with specific plugins or not.

sounds great.

[MVV]

I found that during unpacking only the lower bar representing the total progress is updated, the upper bar for the per-file-progress never gets updated. I think this is by design and fine with me. Though when testing the original 32-bit plugins with Total Commander 32-bit I found, that some of the packer plugins do not draw the upper per-file-progress bar at all. Maybe that's an detail you want to add to your wrapper.

Please tell me which plugin really draws file progress bar and provide me a download link to an example of archive file (I suppose it should be some big file or a slow archiver), I'll check it.

The unpacking process now runs definitely faster with the latest release v1.0.0.194 of your wrapper, compared to the previous release v1.0.0.172. Though unpacking with the wrapper takes about 7-8 times longer compared to the plugins under TC 32-bit.

I think it is because archive contains 4 thousands of files and TC have to call ReadHeader and ProcessFile for every file for unpacking it (and it is critical for unpacking thousands of small files because I can't omit none of these calls like I do with progress callbacks).
I'll try to speed function selection up on the server side.

Also I found that when trying to access any 32-bit plugin config menu via the Pack file dialog, it stalls TC and eventually leads to a crash.

Configuring packer works fine for Total7zip so call is passed. But it doesn't work for some other plugins, I don't know why but I'll try to find a reason.
I'm testing HA plugin and fo some strange reason DialogBox function completely hangs when it is called by the plugin via Wcx6432 server (dialog procedure doesn't called at all). I currently have no guess why it happens.

[Horst.Epp]

Still works fine with my msi test files

Good news.

Do you plan to also add support for WLX plugins ?
I'm interested to get the Syn2 Lister plugin in x64.

Can't you use 64-bit version of Syn2? Or it is another plugin?

Unfortunately the x64 version is not changed since 2012
while the x86 plugin and editor have undergone a lot of improvements.
I had installed the x64 version and had problems so I stopped using it.

[deus-ex]

Please tell me which plugin really draws file progress bar...

From the plugins I tried none appears to be actually using the file progress bar.

I think it is because archive contains 4 thousands of files...

Yes, that's right, but it appears that the process takes unnecessary long. Maybe working with file lists, like in MultiArc for speeding up of batch unpacking, is an option?

[MVV]

Unfortunately WCX interface is quite bad in such sutiation. You have to call ReadHeader* and ProcessFile* up to 4000 times to extract 1 file from this archive. And each call (with corresponding ReadHeader call) have to be passed to archiver for skipping unused files.

However it also seems that archive is listed again and again for every unpacked file, so extracting of 10 files requires about 40000 ProcessFile calls:

Code: Select all

PackSetDefaultParams called
PkSetCryptCallback called
CanYouHandleThisFileW('...\hexen.wad') -> 1
OpenArchiveW -> 0
SetProcessDataProc called
SetChangeVolProc called
SetProcessDataProcW called
SetChangeVolProcW called
ReadHeaderExW -> 10
CloseArchive -> 0
OpenArchiveW -> 0
SetProcessDataProc called
SetChangeVolProc called
SetProcessDataProcW called
SetChangeVolProcW called
ProcessFileW(2, '...\ADVISOR') -> 0 (4254 PK_SKIPs, 4254 total function calls)
ProcessFileW(2, '...\AXE5') -> 0 (3998 PK_SKIPs, 8252 total function calls)
ProcessFileW(2, '...\BALLS1') -> 0 (4081 PK_SKIPs, 12333 total function calls)
ProcessFileW(2, '...\ART1') -> 0 (4125 PK_SKIPs, 16458 total function calls)
ProcessFileW(2, '...\ARTACT1') -> 0 (4129 PK_SKIPs, 20587 total function calls)
ProcessFileW(2, '...\BATS') -> 0 (4136 PK_SKIPs, 24723 total function calls)
ProcessFileW(2, '...\BELLRNG') -> 0 (4146 PK_SKIPs, 28869 total function calls)
ProcessFileW(2, '...\ARMOREXP') -> 0 (4149 PK_SKIPs, 33018 total function calls)
ProcessFileW(2, '...\ACID5') -> 0 (4165 PK_SKIPs, 37183 total function calls)
ProcessFileW(2, '...\BIRD') -> 0 (4174 PK_SKIPs, 41357 total function calls)
CloseArchive -> 0

(counters here show how many times function was called with PK_SKIP action between PK_EXTRACT ones, and just total function calls since plugin loading)

It would be much faster if TC could enumerate archive once for extracting all selected files... I don't understand how TC resets enumeration to the beginning in such scheme. Or maybe it is archiver who resets enumeration after every extracted file? According to logs, after each processed file ReadHeader have began returning names from the beginning.

[MVV]

I've returned hot activity periods (100ms after previous message only Sleep(0) is called, then event waitings work again) to reduce timeouts between function calls and did some other changes but it didn't help a lot.
Native plugin is still 10 times faster (I only extract F folder and native plugin takes about 5 seconds while wrapped one about 50 seconds) in extracting files from this archive.

Wcx6432 Beta 212

[TW]

i tested the msi-plus again, and unpacking all "Files" from calibre-2.58.0.msi runs at the same speed as native plugin x86.

what archive are you testing?

[deus-ex]

...it didn't help a lot.

At least you gained an improvement. Unpacking Doom.wad with v1.0.0.194 takes ~170 seconds, while v1.0.1.212 needs ~145 seconds (native 32-bit plugin needs 22 seconds).

Probably the ball now lies in Christian's corner and he is able to speed up the plugin interface?
Thank you very much for your effort, MVV, it's very much appreciated.

[deus-ex]

i tested the msi-plus again, and unpacking all "Files" from calibre-2.58.0.msi runs at the same speed as native plugin x86.

Yes, the MSI-plus plugin runs about the same speed natively and in the wrapper.

what archive are you testing?

The correct question is: "What plugin are we testing?": GAUP_pro with DooM and Hexen game data files.

[MVV]

Probably the ball now lies in Christian's corner and he is able to speed up the plugin interface?

I'm not sure that Christian can do something with it, I've checked with e.g. MSI-Plus, TC unpacks files in a single pass, so I think that it is GAUP who resets enumeration after every single unpacked file.

i tested the msi-plus again, and unpacking all "Files" from calibre-2.58.0.msi runs at the same speed as native plugin x86.

what archive are you testing?

Please read previous discussion. When files are relatively big, internal plugin speed is the major factor, but with a lot of very small files wrapper overhead is much more noticeable and cause real slowdowns.

deus-ex have mentioned test file Hexen.wad (I've found it somewhere in the Internet and downloaded) which contains more than 4000 files and unpacking every single file requires enumerating files from first file in the archive, so unpacking 10 files may require up to 40000 ReadHeader calls and same number of ProcessFile calls.