Rare problem with ZIP archives

[MarcinW]

I found (by accident) some rare problem with ZIP archives. This is rather special case, but may be a sign of some ZIP handling problem.

There are two ZIP files below (test1.zip and test2.zip), MIME-encoded:

Code: Select all

MIME-Version: 1.0
Content-Type: application/octet-stream; name="test1.zip"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="test1.zip"

UEsDBBQAAgAIAJolKkERzYLtBgAAAAQAAAAIAAAAZmlsZS50eHRLTEpOAQBQSwECFAAUAAIACACa
JSpBEc2C7QYAAAAEAAAACAAAAAAAAAABACAAAAAAAAAAZmlsZS50eHRQSwUGAAAAAAEAAQA2AAAA
LAAAAHh4U09NRSBBRERJVElPTkFMIElORk8gQURERUQgQVQgVEhFIEVORCBYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWCBFTkQgT0YgQURESVRJT05BTCBJTkZP

Code: Select all

MIME-Version: 1.0
Content-Type: application/octet-stream; name="test2.zip"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="test2.zip"

UEsDBBQAAgAIAJolKkERzYLtBgAAAAQAAAAIAAAAZmlsZS50eHRLTEpOAQBQSwECFAAUAAIACACa
JSpBEc2C7QYAAAAEAAAACAAAAAAAAAABACAAAAAAAAAAZmlsZS50eHRQSwUGAAAAAAEAAQA2AAAA
LAAAADMGU09NRSBBRERJVElPTkFMIElORk8gQURERUQgQVQgVEhFIEVORCBYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWCBFTkQgT0YgQURESVRJT05BTCBJTkZP

Important: both files are valid, Files -> Test Archives returns successfully.
Both files are almost identical - there are only 2 bytes of difference.

At the end of each file there is some content added - I found such a file
somewhere in the Internet. I replaced this additional content by "X"-es.

Now we can try to delete zipped file (file.txt) from inside test1.zip.
Everything goes OK. We get empty 22-byte ZIP archive, with additional info removed.

But when we try this with test2.zip, we immediately get an error: "Error in
packed file C:\test2.zip". As a result, we get invalid test2.zip, which contains
only the additional content.

Additional info: When there are more files inside ZIP archive, the additional
info is being preserved when removing files from inside of the archive, but is
being removed when adding files to the archive. The problem appears only when
the last file from the archive is being deleted. Tested also with clean
wincmd.ini.

Regards

[karlchen]

Hello, MarcinW.

Hm, I get different results using (un)zip v6.0 and 7z v9.20 commandline programmes.

unzip v6.0 seems to "like" both files.
7z v9.20 likes test2.zip, but does not like test1.zip from the start. (Can not open file as archive)

zip v3.0 refuses to delete file.txt from test1.zip: zip error: Unexpected end of zip file (test1.zip)
zip v3.0 deletes file.txt from test2.zip, just warning that the archive is empty now. The remaining test2.zip has got 1609 bytes and holds the trailing info.

7z v9.20 refuses to delete file.txt from test1.zip: Error: test1.zip is not supported archive
7z v9.20 deletes file file.txt from test2.zip. Again the remaining file is 1609 bytes large and only holds the additional info.

The fact that 7z and (un)zip yield different results from Total Commander's internal (un)packer raises the question:
Who is right? Who is wrong?
Are the 2 ZIP-files valid ZIP-files at all? I.e. do they conform to the current ZIP-file standard?
If they are not, what is the right way of handling them in this case?

Kind regards,
Karl
--
Environment:

# zip
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
Zip 3.0 (July 5th 2008). Usage:

# unzip
UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP.

# 7z
7-Zip 9.20 Copyright (c) 1999-2010 Igor Pavlov 2010-11-18
p7zip Version 9.20 (locale=de_DE.UTF-8,Utf16=on,HugeFiles=on,4 CPUs)

# lsb_release -d
Description: Linux Mint 12 Lisa
# uname -a
Linux <hostname> 3.0.0-25-generic #41-Ubuntu SMP Mon Aug 13 17:59:54 UTC 2012 i686 i686 i386 GNU/Linux

[MarcinW]

Hello karlchen, thanks for your tests.

There are two possibilities:

1) test1.zip and/or test2.zip are invalid (not conforming to ZIP standard) and each program tries to handle this situation using some non-standard (not described by ZIP standard) algorithms.

2) test1.zip and/or test2.zip are valid (conforming to ZIP standard), but programs are not perfect and can't handle these files properly.

Probably the only way to settle this is to compare test1.zip and test2.zip with the official ZIP file format specification (manually, using hex editor, for example):

http://www.pkware.com/documents/APPNOTE/APPNOTE-6.3.2.TXT

Regards!

[ZoSTeR]

Those are Zip file comments and they are in the specs.

Using pkzipc.exe you can add them with either -header for the whole archive (like in this case) or with -comment for individual files.

7-Zip 9.20 shows them in the "comment" column or in the file properties.

[ghisler(Author)]

TC does not currently support ZIP comments. However, it should preserve them just fine when there are files with them, and the comment length is stored correctly in the headers. I will check the files in the debugger, maybe some field is set incorrectly.