Page 1 of 1

Use libdeflate for ZIP compression

Posted: 2022-12-06, 11:33 UTC
by abouchez
The libdeflate library may be worth investigating, as a complement/alternative/option to the regular zlib.
https://github.com/ebiggers/libdeflate

It is much faster than the regular deflate, and easy to work with.
The only caveat is that it works only on memory buffers.
So the regular stream-oriented zlib is needed for huge .zip but most of the .zip which can stay in memory could benefit from using libdeflate.

See what we did in mORMot:
https://github.com/synopse/mORMot2/blob/master/src/core/mormot.core.zip.pas
and
https://github.com/synopse/mORMot2/blob/master/src/lib/mormot.lib.z.pas

Re: Use libdeflate for ZIP compression

Posted: 2022-12-07, 09:31 UTC
by ghisler(Author)
That looks promising, especially the memory buffers interface would make it very easy to integrate because I'm doing all the other ZIP archive handling functions internally. It seems to rely on advanced CPU features like AVX, so it would have to be implemented in addition to the internal packer.

Re: Use libdeflate for ZIP compression

Posted: 2022-12-07, 16:35 UTC
by abouchez
IIRC libdeflate detects the CPU features, so is able to adapt to the actual CPU it runs on at runtime.

Thanks for the feedback, and looking forward to see it in TC ! :)

Re: Use libdeflate for ZIP compression

Posted: 2022-12-13, 07:29 UTC
by Fla$her
2ghisler(Author)
What about the number of fast bytes?

Re: Use libdeflate for ZIP compression

Posted: 2022-12-13, 16:06 UTC
by ghisler(Author)
That's for the 7zip deflate compressor.

Re: Use libdeflate for ZIP compression

Posted: 2022-12-13, 16:10 UTC
by Fla$her
And it can be implemented with TC7Z*.DLL?

Re: Use libdeflate for ZIP compression

Posted: 2022-12-13, 16:12 UTC
by ghisler(Author)
I'm not using TC7Z*.DLL for ZIP compression, just to unpack 7zip archives. No 7zip archive creation is included in TC, you need a plugin like Total7zip.

Re: Use libdeflate for ZIP compression

Posted: 2022-12-13, 16:25 UTC
by Fla$her
But the description on the link concerns ZIP, not 7ZIP. 7ZIP doesn't have such an option (-mfb).

Re: Use libdeflate for ZIP compression

Posted: 2022-12-14, 08:58 UTC
by ghisler(Author)
It concerns ZIP files generated by 7zip. It only applies to 7zip's internal zip compression function, which isn't used by Total Commander.

Re: Use libdeflate for ZIP compression

Posted: 2022-12-15, 09:12 UTC
by Fla$her
I think if the author was able to implement this only for ZIP, it's only because the Deflate mechanism itself allows it. Don't you think so?
I use -mfb 258 in my codes, which allows me to compress text data better.