Use libdeflate for ZIP compression

Here you can propose new features, make suggestions etc.

Moderators: Stefan2, white, Hacker

Post Reply
abouchez
Junior Member
Junior Member
Posts: 3
Joined: 2022-12-06, 11:26 UTC

Use libdeflate for ZIP compression

Post by *abouchez »

The libdeflate library may be worth investigating, as a complement/alternative/option to the regular zlib.
https://github.com/ebiggers/libdeflate

It is much faster than the regular deflate, and easy to work with.
The only caveat is that it works only on memory buffers.
So the regular stream-oriented zlib is needed for huge .zip but most of the .zip which can stay in memory could benefit from using libdeflate.

See what we did in mORMot:
https://github.com/synopse/mORMot2/blob/master/src/core/mormot.core.zip.pas
and
https://github.com/synopse/mORMot2/blob/master/src/lib/mormot.lib.z.pas
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 44426
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Use libdeflate for ZIP compression

Post by *ghisler(Author) »

That looks promising, especially the memory buffers interface would make it very easy to integrate because I'm doing all the other ZIP archive handling functions internally. It seems to rely on advanced CPU features like AVX, so it would have to be implemented in addition to the internal packer.
Author of Total Commander
https://www.ghisler.com
abouchez
Junior Member
Junior Member
Posts: 3
Joined: 2022-12-06, 11:26 UTC

Re: Use libdeflate for ZIP compression

Post by *abouchez »

IIRC libdeflate detects the CPU features, so is able to adapt to the actual CPU it runs on at runtime.

Thanks for the feedback, and looking forward to see it in TC ! :)
Fla$her
Power Member
Power Member
Posts: 790
Joined: 2020-01-18, 04:03 UTC

Re: Use libdeflate for ZIP compression

Post by *Fla$her »

2ghisler(Author)
What about the number of fast bytes?
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 44426
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Use libdeflate for ZIP compression

Post by *ghisler(Author) »

That's for the 7zip deflate compressor.
Author of Total Commander
https://www.ghisler.com
Fla$her
Power Member
Power Member
Posts: 790
Joined: 2020-01-18, 04:03 UTC

Re: Use libdeflate for ZIP compression

Post by *Fla$her »

And it can be implemented with TC7Z*.DLL?
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 44426
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Use libdeflate for ZIP compression

Post by *ghisler(Author) »

I'm not using TC7Z*.DLL for ZIP compression, just to unpack 7zip archives. No 7zip archive creation is included in TC, you need a plugin like Total7zip.
Author of Total Commander
https://www.ghisler.com
Fla$her
Power Member
Power Member
Posts: 790
Joined: 2020-01-18, 04:03 UTC

Re: Use libdeflate for ZIP compression

Post by *Fla$her »

But the description on the link concerns ZIP, not 7ZIP. 7ZIP doesn't have such an option (-mfb).
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 44426
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Use libdeflate for ZIP compression

Post by *ghisler(Author) »

It concerns ZIP files generated by 7zip. It only applies to 7zip's internal zip compression function, which isn't used by Total Commander.
Author of Total Commander
https://www.ghisler.com
Fla$her
Power Member
Power Member
Posts: 790
Joined: 2020-01-18, 04:03 UTC

Re: Use libdeflate for ZIP compression

Post by *Fla$her »

I think if the author was able to implement this only for ZIP, it's only because the Deflate mechanism itself allows it. Don't you think so?
I use -mfb 258 in my codes, which allows me to compress text data better.
Post Reply