Internal TAR Archiver corrupts TARs if they are too big
Moderators: Hacker, petermad, Stefan2, white
Internal TAR Archiver corrupts TARs if they are too big
I tested it with v9.0a 32bit on Windows 10 64bit and the Bug is serious because the size of the TAR itself match but if you enter the TAR most files and directories are missing.
Also restoring the whole TAR doesn't help either, the files and directories that was missing in the listing are also missing on restore, so those files and directories are definitely lost.
It happens if the TAR gets to big, like around 45GB, it's easy to test and happens always.
I also tested it with 7z and store only and this works fine, nothing missing.
So beware using Total Commanders TAR for backups or similar if the size gets ~45GB+ the size of the TAR will match but files and whole directories will be lost, so this bug is a bit dangerous.
I won't use Total Commanders TAR anymore, and use 7z store instead till this is fixed.
Also restoring the whole TAR doesn't help either, the files and directories that was missing in the listing are also missing on restore, so those files and directories are definitely lost.
It happens if the TAR gets to big, like around 45GB, it's easy to test and happens always.
I also tested it with 7z and store only and this works fine, nothing missing.
So beware using Total Commanders TAR for backups or similar if the size gets ~45GB+ the size of the TAR will match but files and whole directories will be lost, so this bug is a bit dangerous.
I won't use Total Commanders TAR anymore, and use 7z store instead till this is fixed.
- ghisler(Author)
- Site Admin
- Posts: 49624
- Joined: 2003-02-04, 09:46 UTC
- Location: Switzerland
- Contact:
How was the TAR created? With Total Commander? Some other tool? If TC did you check the option "Create TAR files for Linux"?
Author of Total Commander
https://www.ghisler.com
https://www.ghisler.com
It was created using the TC internal TAR and i tried it with "Create TAR files for Linux" on and off, doesn't change anything.
Also to reproduce it i tested it with an a gamefolder that is ~45GB and should be the same for all that have it the Dishonored 2 steam folder.
The resulting TAR is 45GB, the content only shows ~7.6GB and if the 45GB TAR is unpacked as whole only ~7.6Gb is unpacked, but it goes trough the whole 45GB.
Packing the same folder with 7z plugin and choosing store only the resulting archive is 45GB and all content is visible and also unpacked successfully.
Also to reproduce it i tested it with an a gamefolder that is ~45GB and should be the same for all that have it the Dishonored 2 steam folder.
The resulting TAR is 45GB, the content only shows ~7.6GB and if the 45GB TAR is unpacked as whole only ~7.6Gb is unpacked, but it goes trough the whole 45GB.
Packing the same folder with 7z plugin and choosing store only the resulting archive is 45GB and all content is visible and also unpacked successfully.
- ghisler(Author)
- Site Admin
- Posts: 49624
- Joined: 2003-02-04, 09:46 UTC
- Location: Switzerland
- Contact:
I have just created a 53 GB TAR file for test purposes, containing 207 large files in various subfolders. Then I unpacked it and compared the results with the original files (by content). All files were completely unpacked, and there were no differences to the originals.
Apparently something must be happening on your system around that 7.6GB mark. For example, it could be a problematic file name (too long, bad name like nul or so), or the system could be running out of file handles.
It could also be the number of files you pack. Could you please enter your TAR file and press Ctrl+B, then select one file to see the total number of files?
Apparently something must be happening on your system around that 7.6GB mark. For example, it could be a problematic file name (too long, bad name like nul or so), or the system could be running out of file handles.
It could also be the number of files you pack. Could you please enter your TAR file and press Ctrl+B, then select one file to see the total number of files?
Author of Total Commander
https://www.ghisler.com
https://www.ghisler.com
Re: Internal TAR Archiver corrupts TARs if they are too big
The bug exists and can be reproduced in version 9.51. The size of tar archive doesn't matter. The bug occurs when size of file being archived is over 2 GB. Every file in archive after such large file is inaccessible. This large file (> 2GB) is the last file that can be unpacked, all files after it are not visible. In other words, tar archive works fine as long as all files are less then 2 GB or there is only one file greater than 2 GB and this large file is the last one in archive. To reproduce you only need two files: one over 2 GB and any other. The large file has to be packed into archive first (for example it can be named "1.bin" and the other one "2.bin"). You can also just pack two files, each over 2 GB. When unpacking such archive only first file will be accessible, second won't be visible. I can see contents of both files in the archive (in binary viewer). I believe all files are actually packed into archive, but large file (> 2GB) cause archive header corruption. You should be able to recover files (using external utility) if you can find their offset and length in the archive.
Re: Internal TAR Archiver corrupts TARs if they are too big
Confirmeddajar wrote: ↑2020-07-13, 18:37 UTC The bug exists and can be reproduced in version 9.51. The size of tar archive doesn't matter. The bug occurs when size of file being archived is over 2 GB. Every file in archive after such large file is inaccessible. This large file (> 2GB) is the last file that can be unpacked, all files after it are not visible. In other words, tar archive works fine as long as all files are less then 2 GB or there is only one file greater than 2 GB and this large file is the last one in archive. To reproduce you only need two files: one over 2 GB and any other. The large file has to be packed into archive first (for example it can be named "1.bin" and the other one "2.bin"). You can also just pack two files, each over 2 GB. When unpacking such archive only first file will be accessible, second won't be visible. I can see contents of both files in the archive (in binary viewer). I believe all files are actually packed into archive, but large file (> 2GB) cause archive header corruption. You should be able to recover files (using external utility) if you can find their offset and length in the archive.
Windows 11 Home, Version 24H2 (OS Build 26100.2605)
TC 11.50RC1 x64 / x86
Everything 1.5.0.1387a (x64), Everything Toolbar 1.5.1, Listary Pro 6.3.1.84
QAP 11.6.4 x64
TC 11.50RC1 x64 / x86
Everything 1.5.0.1387a (x64), Everything Toolbar 1.5.1, Listary Pro 6.3.1.84
QAP 11.6.4 x64
- ghisler(Author)
- Site Admin
- Posts: 49624
- Joined: 2003-02-04, 09:46 UTC
- Location: Switzerland
- Contact:
Re: Internal TAR Archiver corrupts TARs if they are too big
I will check it.
Author of Total Commander
https://www.ghisler.com
https://www.ghisler.com
- ghisler(Author)
- Site Admin
- Posts: 49624
- Joined: 2003-02-04, 09:46 UTC
- Location: Switzerland
- Contact:
Re: Internal TAR Archiver corrupts TARs if they are too big
Thanks, I can reproduce it now. TC adds an extra 512 byte header (all zeroes) to the end of files >2GB but <4GB. In the next version, I will fix thsis problem and also make it possible to read and unpack archives with this problem. Please contact me by e-mail if you need a pre-release version to access your archives.
Author of Total Commander
https://www.ghisler.com
https://www.ghisler.com