[WCX] ZPAQ

[milo1012]

Some additional info concerning the latest bug:
It is a real code optimization bug caused by the /Og compiler option (global optimizations),
which is either enabled directly, or by using the default optimization level /O2 (and /O1 too).

Sure, I could disable optimizations, but this would make the plug-in quite slow.

Like I said, I tested the bug to be present in Visual Studio 2010 (SP1) and VS 2012 (Update 5).
Unfortunately I don't have a license for Visual Studio 2013/2015 and no time to deal with the express editions.
It would be great if someone could compile the 64-bit dll with one of those compilers and send it to me, or at least test them for this bug and report to me.
(source for 1.3.5 - the project should convert trouble-free, and just do a simple compile to x64 ("Configuration manager"))
If this issue would still be present for those newest compilers, we probably should send a bug report to Microsoft, as this is quite serious.

In any case, I will compile the next release with VS 2008 and reimplement the ZPAQ 7.07 code.
I also found a way to send some of the status updates mentioned by ykhabins to the progress dialog (it might not always work though).

[Skif_off]

2milo1012
zpaq.lng with Russian translation.

[TW]

i tried to compile it with GCC, but no luck.

[milo1012]

zpaq.lng with Russian translation.

Many thanks!

i tried to compile it with GCC, but no luck.

Thanks for trying.
Indeed, there were some errors, due to pedantic language standards.
I fixed some of them, and I was able to make it compile in TDM-GCC.

I still won't release the plug-in compiled with MinGW.
Reason: it links to msvcrt.dll, where I can't catch memory errors.
When there are errors due to memory shortage (32-bit), it will most likely crash the whole TC process,
because msvcrt.dll won't find a target memory buffer.
With a VS compile, catching memory exceptions will work most of the time, and you can continue work in TC.

[TW]

ah, got it, thanks.

[milo1012]

2Ghisler
I recently experienced crashes while sending a lot of updates to the progress dialog while packing.
But the crash seems to happens after the PackFilesW() procedure finished, and all my memory was cleared correctly.
But after that, TC sometimes (in maybe one out of ~5 passes) gets unstable.
Sometimes I get a freeze, sometimes I get:

Code: Select all

---------------------------
Total Commander 8.52a
---------------------------
Fatal error in zip thread, aborting!
External exception C0000008
Windows 7 SP1 6.1 (Build 7601)

Please report this error to the Author, with a description
of what you were doing when this error occurred!

Windows exception: C0000008
Stack trace:
77912A4C
70619E  5B9DAF  5BD07C  >418CBC  40362C  
Raw:
70619E  704B2F  703703  703AC5  466034  6DFD12
560051  5B9DAF  689431  68943B  5BD07C  418CBC
40362C  
Press Ctrl+C to copy this report!
---------------------------
OK   
---------------------------

Sometimes even:

Code: Select all

---------------------------
Total Commander 8.52a
---------------------------
Fatal error in zip thread, aborting!
External exception C0000008
Windows 7 SP1 6.1 (Build 7601)

Please report this error to the Author, with a description
of what you were doing when this error occurred!

Windows exception: C0900008
Stack trace:

Raw:


Press Ctrl+C to copy this report!
---------------------------
OK   
---------------------------

(yes, no stack trace)


After confirming the dialog, TC seems to be sometimes stuck in one thread.
PE thread stack:

Code: Select all

ntoskrnl.exe!KeSetEvent+0x6ca
ntoskrnl.exe!KeWaitForMultipleObjects+0xd52
ntoskrnl.exe!KeWaitForSingleObject+0x19f
ntoskrnl.exe!PoStartNextPowerIrp+0xbb4
ntoskrnl.exe!PoStartNextPowerIrp+0x1841
ntoskrnl.exe!PoStartNextPowerIrp+0x1ab7
ntdll.dll!RtlReleaseSRWLockShared+0x14
ntdll.dll!RtlMoveMemory+0x571
kernel32.dll!GetProfileStringW+0x26b
ntdll.dll!RtlReleaseSRWLockShared+0x14
ntdll.dll!RtlMoveMemory+0x3dc
ntdll.dll!RtlMoveMemory+0x2f2
ntdll.dll!RtlUnwind+0xd9
kernel32.dll!GetProfileStringW+0x26b
kernel32.dll!RegSaveKeyExW+0x558
ntdll.dll!RtlMoveMemory+0x55d
ntdll.dll!RtlMoveMemory+0x52f
ntdll.dll!KiUserExceptionDispatcher+0xf
TOTALCMD.EXE+0x6fc1c
TOTALCMD.EXE+0x70000
TOTALCMD.EXE+0x30e6
ntdll.dll!RtlMoveMemory+0x52f
ntdll.dll!KiUserExceptionDispatcher+0xf
ntdll.dll!LdrFindResource_U+0x5d6
ntdll.dll!KiUserCallbackDispatcher+0x2e
TOTALCMD.EXE+0x3f515
TOTALCMD.EXE+0x3af54
TOTALCMD.EXE+0x4b16b
TOTALCMD.EXE+0x48be5
TOTALCMD.EXE+0x4b13d
TOTALCMD.EXE+0x487d4
TOTALCMD.EXE+0x25bc2
TOTALCMD.EXE+0x487f7
TOTALCMD.EXE+0x47d79
TOTALCMD.EXE+0x25bc2
TOTALCMD.EXE+0x47d79
ntdll.dll!KiUserCallbackDispatcher+0x2e
TOTALCMD.EXE+0x2b2a6
TOTALCMD.EXE+0x2b342
TOTALCMD.EXE+0x306b0d
kernel32.dll!BaseThreadInitThunk+0x12
ntdll.dll!RtlInitializeExceptionChain+0x63
ntdll.dll!RtlInitializeExceptionChain+0x36

Any idea what might cause this?
(maybe the progress dialog accessed after PackFilesW() finished?)

[ghisler(Author)]

The error occurs AFTER TC has closed the progress dialog, when it tries to show a MessageBox with an error. The error itself isn't in my code, but somewhere in MessageBoxW.

I guess that you are trying to send something to the progress dialog when it's already gone. Try using IsWindow() before sending your message.

[milo1012]

Try using IsWindow() before sending your message.

Thanks!
I will try that.

Though it's strange, as all dialog updates should be ceased until then.

[milo1012]

Try using IsWindow() before sending your message.

I just realized:
how am I supposed to use IsWindow() when I only have a function pointer for the dialog update (received via SetProcessDataProc()) ?
How would I get a Window handle? Enumerating all process Windows?

Anyway, I'm really sure that I don't cause the bug directly, as I don't do anything wrong per se.
All dialog updates definitively cease, before PackFiles() returns.
I counter-checked with a global log and even a different compiler.
The last progress dialog update will be finished before any other thing.

I forgot to mention in my report above that of course I returned an error code in PackFilesW() (E_EABORTED in this case).
But when I return zero from PackFilesW() (i.e. no error), there will be no crash, no matter how often I try,
in opposite to returning any error code, where I get a crash every couple of test runs.
So it seems to involve TC's MessageBox.

Interestingly, when I show my own MessageBoxW() before returning from PackFilesW(), there is no crash at all, even after the 100th try.
But when I do a simple Sleep() before returning, there will be a crash every couple of runs, no matter for how long I use Sleep().

So I guess that it's some strange issue when calling certain Api functions in a certain order.

Meanwhile I got a different stack trace BTW:

Code: Select all

---------------------------
Total Commander 8.52a
---------------------------
External exception C0000008.
External exception C0000008
Windows 7 SP1 6.1 (Build 7601)

Please report this error to the Author, with a description
of what you were doing when this error occurred!

Windows exception: C0000008
Stack trace:
77912B59
43F2D6  44B169  448BE3  44B13B  4487D2  >425BC0
447D77  425BC0  42B2A4  42B340  706B0B  
Raw:
425BC0  449851  446865  448A09  43E7C7  43F2D6
43AEFD  43F069  43F499  44B169  448BE3  44B13B
446865  448AA1  447E42  4487D2  425BC0  4487F5
447D77  425BC0  447D77  42B2A4  42B340  42B4FA
706B0B  
Press Ctrl+C to copy this report!
Continue execution?
---------------------------
Ja   Nein   
---------------------------

And:

Code: Select all

---------------------------
Total Commander 8.52a
---------------------------
Fatal error in zip thread, aborting!
External exception C0000008
Windows 7 SP1 6.1 (Build 7601)

Please report this error to the Author, with a description
of what you were doing when this error occurred!

Windows exception: C0000008
Stack trace:
77912B59

Raw:
61687D  61926B  >655179  446865  4487D2  425BC0
447D77  
Press Ctrl+C to copy this report!
---------------------------
OK   
---------------------------

I don't know how I would fix such thing.
But I think that the crash is still rare, so it's probably not that bad in practice.

[milo1012]

New Version 1.4!

• reimplemented ZPAQ 7.07 core
• the block size for compression can now be set in the config dialog
  -> currently limited to 64 MiB for any compression level (for safety reasons and for keeping
  the rough memory requirements, the calculations and warnings in reasonable limits)
  -> decompressing archives created with larger block sizes (standalone zpaq) will work,
  but for a huge size the memory requirements might be impossible for the 32-bit plug-in
  (no way to detect actual requirements before starting decompression)
• when packing, the progress dialog now receives the same amount of updates as standalone zpaq
  -> packing huge single files now might show intermediate stages (and not just
  zero and 100 percent)
  -> amount of steps depends on the block size VS deduplicated file size
• canceling a pack process can now optionally discard the current archive update (new option)
  -> the partially appended data will be ignored when reading the archive and will be overwritten
  on the next update
  -> canceling a pack process can trigger faster with this option (on block instead of file boundaries)
• when packing, the progress dialog can now optionally show the same block stats as standalone zpaq
  -> format '(XX,XXX,X)': compression level and block size, estimated compressibility
  (0..255, where 0 means random), detected type (1 for text, 2 for x86, 3 for both, or 0 for neither)
  -> might trigger wrong "file count" stats in the dialog (depending on block size)
• added Russian translation (by 'Skif_off')
• some internal code improvements to make the source compile in GCC

Check the first post for the new file.



Believe it or not, for some reason the newest compile in VS 2010 works just as with < 1.3.5.
No bugs found, after an intense testing.
I guess the changes I made to the add() routine somehow triggered a correct loop for the optimizer now,
which would mean that with 1.3.5 the loop was logically incorrect for the optimizer (but it worked flawless for 32-bit) !?
The pleasures of an optimizing compiler...

To be honest, I don't want to know
I have my share of experience when it comes to compilation bugs, but this is the most obscure so far.


Oh and BTW, when compiling the x64 release in VS 2008, performance is horrible (down at least 50 percent), and I don't want to know the reason for that as well


Also, if there are no critical bugs in this version, this will be the last major release for a longer period of time,
as I have different things to attend to during the next months.

[Skif_off]

2milo1012
zpaq.lng with Russian translation.

[petermad]

2milo1012

Danish translation for version 1.4 can now be downloaded at: http://madsenworld.dk/tcmd/wcx_zpaq_14_danish.zip

[milo1012]

2Skif_off and petermad
Thank you both!

I'll wait for other things to add before I release a minor new version with just an updated language file.
In the meantime you can download the combined lng file at:
http://wincmd.ru/files/9924355/zpaq_lng_14.zip

[ghisler(Author)]

how am I supposed to use IsWindow() when I only have a function pointer for the dialog update (received via SetProcessDataProc()) ?

Sorry, your bug report sounded as if you were sending messages to the window, not using the processdataproc.

Are you calling the callback after the file was closed? What exactly are you passing via callback? And did you define it as stdcall?

[milo1012]

Are you calling the callback after the file was closed?

I'm using a structure on heap with all data, including a handle to the file.
I'm sure that the file and handle will be closed when I delete the structure (C++ destructor),
which I always do before I return from PackFiles().
Therefore: No, I don't think that I keep the file open or locked in any way when PackFiles() returns,
and all function calls through the callback function definitely ceased before that.
(as all calls happen on the main packer thread only, from which I return from PackFiles() of course)

What exactly are you passing via callback?

Only what the function permits, i.e. tProcessDataProcW with a WCHAR* and an int.
I use a two-line string at some point, i.e. text\r\ntext ,
but this doesn't seem to be a problem or the cause of it.

And did you define it as stdcall?

Yes, I'm using the standard C headers defined in the wcx guide.
I think it wouldn't work at all if this wouldn't be the case.


Like I said, the crash is rare, and only seems to happen when TC tries to show it's own MessageBox.
I know that C0000008 is an invalid handle.
At some point I will try to debug the plugin, but currently I don't have the time for it.