7zip native support

[AntonyD]

So in essence using an external archiver still makes a lot more sense and logic than trying to use some kind of incomplete connection to the 7zip library. This weird highlighting of just ZIP format (but by 7-ZIP) and only 7z (by 7-ZIP) is very unproductive. IMHO.

[lelik007]

2ghisler(Author)
Other bugs and mismatches found by the other users.

4. Critical bug - If you put -v with some value like -v1m or -v10m in Extra Parameters for the internal packer TC creates a lot of volumes of zero sizes and crashes after this (confirmed by me).

5. PPMD algo uses only 1 thread whatsoever and this value is locked in 7-zip.

6. Bzip Dictionaries mismatches, should be:
level 1-2 - 100 Kb
level 3-4 - 500 Kb

[lelik007]

2AntonyD
Different formats have the different switches and the internal logic how they're processed. As you see work with .7z format only is not ideal now, so I think it's not a good idea to start redoing some other format until we're done here.

[ghisler(Author)]

1. OK I will round up
2. I can confirm these values also for 32-bit 7zfm.exe
3. I checked them too, thanks.
4. Confirmed, passing -v manually isn't currently supported. Use the checkbox in the pack dialog
5. I'm currently not using the defaults for threads. The user can set an upper limit, and I use what the algorithm supports
6. Confirmed

[lelik007]

2ghisler(Author)

1. Addition about the mismatches for x64, x32, should be for PPMD by default:

Spoiler

level 2: dictionary size - 2 Mb, word size - 4
level 4: dictionary size - 8 Mb, word size - 4
level 6: dictionary size - 32 Mb, word size - 6
this affects x32 only:
level 8: dictionary size - 128 Mb, word size - 16
level 9: dictionary size - 256 Mb, word size - 32

4. About 4. Bug was found because of TC's help:

The internal packer only supports extra parameters starting with -m, plus the following: -slp (set large page mode), -stl (set archive date to that of the newest file), -v (volume size).

The user just put -v in Extra Parameters because of this sentence.

5.

I'm currently not using the defaults for threads. The user can set an upper limit, and I use what the algorithm supports.

You can set and lock the thread value for PPMD as 1 if it's possible.
---
7-zip uses the maximum thread amount for LZMA2 so it's more safe what's in TC. Because:
level 8-9 with 8 threads and 256 Mb dictionary requires 11,5 Gb RAM and 6 Gb RAM for 4 threads.
But for 2 threads which are the defaults in TC it's only less than 3 Gb RAM.

[ghisler(Author)]

1. Dictionary sizes:

this affects x32 only:
level 8: dictionary size - 128 Mb, word size - 16
level 9: dictionary size - 256 Mb, word size - 32

Interesting that it goes to 256 MB with PPMd but not with LZMA*. Probably because of only one thread.

4. Currently TC only supports -v<number> without a multiplier like k or M. Writing -v1m was interpreted as -v1, so 1 Byte per volume.
I will support for -v1m and also -v=1m in the next beta.
I will also abort the operation if the volume size is smaller than 1/1000 of the total unpacked size. Why? It seems to be a bug/misfeature of the 7z.dll: It allocates all the part files while packing, but does not write anything to them yet. It also keeps all these part files open! When the volume size is too small, Win32 will run out of file handles at about 1800 files and crash hard, and win64 at about 8000 files and hang. Limiting the maximum parts to 1000 should prevent that from happening. Unfortunately there are no callbacks for volume creation, so I can't handle the out of file handles situation myself.

5. That's why I leave it to the user to set a higher thread count.

[lelik007]

2ghisler(Author)

Interesting that it goes to 256 MB with PPMd but not with LZMA*. Probably because of only one thread.

1. Actually, no, because PPMd is symmetric this takes almost the same memory size to pack and unpack, level 9 ~ 290 Mb with 256 Mb dictionary.
PPMd is suitable for a plain text only.
Meanwhile LZMA level 9 has bt4 match finder with 1/11 ratio for packing so this takes ~ 2800 Mb to pack and ~ 260 Mb to unpack with 256 Mb dictionary, 2 threads or even 1. And because the ratio is so high the maximum dictionary size for version x32 was set as 64 Mb.

4. You can allow any minimum volume size that you consider safe to use it's just should be mentioned in help.

5. Totally agree.

[ghisler(Author)]

4. There is no minimum which is safe to use, the minimum increases the larger the total size of the files you pack.

I would like to return to the following bug report:
viewtopic.php?p=464334#p464334

Despite all my tries, I could never reproduce this problem. Can anyone else besides lelik007 reproduce it?

I also checked this in the debugger, and tcbit7z only passes cpuz_x64.exe to the compressor to add. It doesn't pass any directory to it.

[lelik007]

2ghisler(Author)
I understood what the difference is: in the video you provided you did it right, but you added just the files to the archive, I added a folder.
Take a look at the right path : https://www.upload.ee/image/17486292/Subfolder_bug.jpg
TC still creates a sub-folder of a folder, but in the video you don't have a folder in that archive at all.

I think this is why you could not reproduce it.
And with the same conditions if "Save Paths" is checked it does and doesn't create if it's unchecked.
---
I added cpuz_x64.exe with the replacement and the result is above.
---
Please, try again, but pack the folder not the files.

[ghisler(Author)]

Thanks, that helped!

[lelik007]

2ghisler(Author)
I checked what I could:
1. Only mismatch is left as I see, Solid Block size 7-zip v24.09 x64, should be:
LZMA:
level 4 - 2 Gb
---
And I think as for PPMd the number of the threads could be set as 1, this is it.
All the other defaults for x64 are correct.

2. TC's memory calculator still shows 1 Gb less than 7-zip.
3. DESCRIPT.ION should be edited:

TC7Z64.DLL 7-Zip unpacker DLL (64-bit)

It's now like: TC7Z64.DLL 7-Zip packer DLL (64-bit) or TC7Z64.DLL 7-Zip packer/unpacker DLL (64-bit).
===
5. The bug with -v: was fixed, I created an archive with 800 volumes by -v1m, each volume is 1 mb.
6. The bug with adding a sub folder of a folder in 7z archive while replacing was fixed. I checked with 3 archives.

[lelik007]

2ghisler(Author)
Christian, as we discussed some time ago an external 7-zip packer can't add a sub-folder to an archive, because 7zG.exe/7z.exe don't have an appropriate switch for this, because 7zFM.exe does it. But at some point you can make an exception, whatever is chosen the internal packer would do this because it's just able to. It's not a FR, rather an idea how it can be done.

[ghisler(Author)]

2lelik007
1. Confirmed, strange that I missed that. 32-bit 7zfm uses the same
2. Not confirmed, for which options?
3. OK
5. I changed the tcbit7z code to only keep one part open at any given time, so it should now work even with many more parts. I set an upper limit of 100000 parts because it really makes no sense to create that many parts. But otherwise even 1 byte parts work!
6. Great, thanks!

2lelik007
You mean I should automatically switch to the internal packer when the user tries to pack to a subdirectory?

[lelik007]

2ghisler(Author)
2. This is just for example, https://www.upload.ee/image/17501594/TC7zCalc.jpg
same if you choose 3072m dictionary, 7-zip shows 32Gb TC - 31 Gb ; same with 5 threads, 7-zip shows 72 Gb, TC - 71 Gb.

You mean I should automatically switch to the internal packer when the user tries to pack to a subdirectory?

Yes, I meant if a user tries to pack to a subdirectory this command could be redirected to the internal packer, but whatever is chosen in GUI stays the same. This because we don't know how to do this with an external packer.
---
And I actually meant you could do this, but not should - this is for your choice.

[ghisler(Author)]

2. It looks like 7-Zip is rounding up GBytes, TC only rounds up MBytes.

Yes, I meant if a user tries to pack to a subdirectory this command could be redirected to the internal packer, but whatever is chosen in GUI stays the same.

OK, I can add this.