Internal TAR Archiver corrupts TARs if they are too big

Please report only one bug per message!

Moderators: white, Hacker, petermad, Stefan2

Post Reply
Nirantali
Junior Member
Junior Member
Posts: 4
Joined: 2017-08-01, 00:12 UTC

Internal TAR Archiver corrupts TARs if they are too big

Post by *Nirantali »

I tested it with v9.0a 32bit on Windows 10 64bit and the Bug is serious because the size of the TAR itself match but if you enter the TAR most files and directories are missing.

Also restoring the whole TAR doesn't help either, the files and directories that was missing in the listing are also missing on restore, so those files and directories are definitely lost.

It happens if the TAR gets to big, like around 45GB, it's easy to test and happens always.

I also tested it with 7z and store only and this works fine, nothing missing.

So beware using Total Commanders TAR for backups or similar if the size gets ~45GB+ the size of the TAR will match but files and whole directories will be lost, so this bug is a bit dangerous.

I won't use Total Commanders TAR anymore, and use 7z store instead till this is fixed.
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48005
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

How was the TAR created? With Total Commander? Some other tool? If TC did you check the option "Create TAR files for Linux"?
Author of Total Commander
https://www.ghisler.com
Nirantali
Junior Member
Junior Member
Posts: 4
Joined: 2017-08-01, 00:12 UTC

Post by *Nirantali »

It was created using the TC internal TAR and i tried it with "Create TAR files for Linux" on and off, doesn't change anything.

Also to reproduce it i tested it with an a gamefolder that is ~45GB and should be the same for all that have it the Dishonored 2 steam folder.

The resulting TAR is 45GB, the content only shows ~7.6GB and if the 45GB TAR is unpacked as whole only ~7.6Gb is unpacked, but it goes trough the whole 45GB.

Packing the same folder with 7z plugin and choosing store only the resulting archive is 45GB and all content is visible and also unpacked successfully.
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48005
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

I have just created a 53 GB TAR file for test purposes, containing 207 large files in various subfolders. Then I unpacked it and compared the results with the original files (by content). All files were completely unpacked, and there were no differences to the originals.

Apparently something must be happening on your system around that 7.6GB mark. For example, it could be a problematic file name (too long, bad name like nul or so), or the system could be running out of file handles.

It could also be the number of files you pack. Could you please enter your TAR file and press Ctrl+B, then select one file to see the total number of files?
Author of Total Commander
https://www.ghisler.com
dajar
New Member
New Member
Posts: 1
Joined: 2020-07-13, 18:26 UTC

Re: Internal TAR Archiver corrupts TARs if they are too big

Post by *dajar »

The bug exists and can be reproduced in version 9.51. The size of tar archive doesn't matter. The bug occurs when size of file being archived is over 2 GB. Every file in archive after such large file is inaccessible. This large file (> 2GB) is the last file that can be unpacked, all files after it are not visible. In other words, tar archive works fine as long as all files are less then 2 GB or there is only one file greater than 2 GB and this large file is the last one in archive. To reproduce you only need two files: one over 2 GB and any other. The large file has to be packed into archive first (for example it can be named "1.bin" and the other one "2.bin"). You can also just pack two files, each over 2 GB. When unpacking such archive only first file will be accessible, second won't be visible. I can see contents of both files in the archive (in binary viewer). I believe all files are actually packed into archive, but large file (> 2GB) cause archive header corruption. You should be able to recover files (using external utility) if you can find their offset and length in the archive.
User avatar
Horst.Epp
Power Member
Power Member
Posts: 6429
Joined: 2003-02-06, 17:36 UTC
Location: Germany

Re: Internal TAR Archiver corrupts TARs if they are too big

Post by *Horst.Epp »

dajar wrote: 2020-07-13, 18:37 UTC The bug exists and can be reproduced in version 9.51. The size of tar archive doesn't matter. The bug occurs when size of file being archived is over 2 GB. Every file in archive after such large file is inaccessible. This large file (> 2GB) is the last file that can be unpacked, all files after it are not visible. In other words, tar archive works fine as long as all files are less then 2 GB or there is only one file greater than 2 GB and this large file is the last one in archive. To reproduce you only need two files: one over 2 GB and any other. The large file has to be packed into archive first (for example it can be named "1.bin" and the other one "2.bin"). You can also just pack two files, each over 2 GB. When unpacking such archive only first file will be accessible, second won't be visible. I can see contents of both files in the archive (in binary viewer). I believe all files are actually packed into archive, but large file (> 2GB) cause archive header corruption. You should be able to recover files (using external utility) if you can find their offset and length in the archive.
Confirmed
Windows 11 Home x64 Version 23H2 (OS Build 22631.3296)
TC 11.03 x64 / x86
Everything 1.5.0.1371a (x64), Everything Toolbar 1.3.2, Listary Pro 6.3.0.69
QAP 11.6.3.2 x64
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48005
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Internal TAR Archiver corrupts TARs if they are too big

Post by *ghisler(Author) »

I will check it.
Author of Total Commander
https://www.ghisler.com
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48005
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Internal TAR Archiver corrupts TARs if they are too big

Post by *ghisler(Author) »

Thanks, I can reproduce it now. TC adds an extra 512 byte header (all zeroes) to the end of files >2GB but <4GB. In the next version, I will fix thsis problem and also make it possible to read and unpack archives with this problem. Please contact me by e-mail if you need a pre-release version to access your archives.
Author of Total Commander
https://www.ghisler.com
Post Reply