Total Commander can't unpack ZSTD files with dictionaries larges than 128 Mb.

[lelik007]

Total Commander can't unpack ZSTD files with dictionaries larges than 128 Mb. It just hangs.
I tested packing with reference utility. Everybody can check it.

https://github.com/facebook/zstd/releases/download/v1.5.5/zstd-v1.5.5-win32.zip
https://github.com/facebook/zstd/releases/download/v1.5.5/zstd-v1.5.5-win64.zip

Just like this

Code: Select all

zstd.exe --long=28 -2 (some file larger than 256 Mb) 

--long can be up to 30 (1Gb for x32) and (2Gb for x64) default is 27 = 128 Mb
--long means 2^27 in bytes. And if you pack some file with --long more than 27 you must pass
this for unpacking

Code: Select all

zstd.exe --long=28 -d (some file larger than 256 Mb) 

--long[=#]
enables long distance matching with # windowLog, if not # is not present it defaults to 27. This increases the window size (windowLog) and memory usage for both the compressor and decompressor. This setting is designed to improve the compression ratio for files with long matches at a large distance.
Note: If windowLog is set to larger than 27, --long=windowLog or --memory=windowSize needs to be passed to the decompressor.

I know this issue - it was in Winrar but Eugene Roshal fixed this in Version 6.20
https://www.rarlab.com/rarnew.htm

8. Added decompression of .zst long range mode archives with dictionary
exceeding 128 MB. Previously it was possible to decompress them only
if dictionary was 128 MB or less.

[ghisler(Author)]

I'm using the official release 1.5.5 of the ZSTD library from here:
https://github.com/facebook/zstd

I have compared the current development code with the release and there are quite a lot of changes, so I cannot say which of them is responsible for the addition of a 256MB dictionary - seems that 1.5.5 doesn't support it yet. I'm reluctant to use the current development code because it's in active development and may contain other bugs.

[lelik007]

2ghisler(Author)
I'm not a coder of any kind and can't tell anything about the code.
But this option is in the reference utility since 1.5.2 - I'm sure.
I used this manual to https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md
to describe what --long is.

[ghisler(Author)]

Maybe it's caused by this option:
-M#, --memory=#: Set a memory usage limit. By default, zstd uses 128 MiB for decompression as the maximum amount of memory the decompressor is allowed to use, but you can override this manually if need be in either direction (i.e. you can increase or decrease it).

Do you have to specify -M256MiB to unpack such archives on the command line?

[ghisler(Author)]

After checking the problem in the debugger, I found out that the dictionary size is artificially limited to 128 MBytes via a constant in the code named ZSTD_WINDOWLOG_LIMIT_DEFAULT.

I have changed that now to 1 GByte (ZSTD_WINDOWLOG_LIMIT_DEFAULT 30) and it seems to work. Here are the updated dlls:
https://www.totalcommander.ch/win/tools/tczstd_256mb.zip
To install, close and re-open Total Commander to make sure the old dlls are unloaded. Then double click on the above zip file in Total Commander to install the new dlls.

[lelik007]

2ghisler(Author)
Yes. You can use either option to unpack such archives.

zstd -d -M2048MB or
zstd -d --long=31 do the job

-M2048MB is max for x64
-M1024MB is max for x32

I tested x64 dll you provided - it works fine for 1 Gb dictionary but hangs with 2 Gb.
You should increase it to 31 for x64 dll but it's 30 max for x32.

[ghisler(Author)]

I didn't see any hints about a 2GB window, only 1GB, so I used that as the maximum. Please re-download the above zip file, it contains an updated 64-bit dll. If you get the same file again, you will have to close and re-open your web browser.
You will have to wait for TC 11.03 to get an error in 32-bit when trying to unpack a file with 2GB window.

[lelik007]

2ghisler(Author)

I didn't see any hints about a 2GB window

I'll show you then.
Image: https://ibb.co/FbXLx9W
This is the reference x64 utility and it says 2048 MB so as Winrar x64 does.

And I asked Eugene Roshal of Winrar for what he did to address the issue - he said literally:
"I called ZSTD_DCtx_setMaxWindowSize(dctx,(1U<<31)); "
Whatever this means.

Thank you Christian, I tested x64 dll and it works fine. As for x32 - I haven't used it for years.

[ghisler(Author)]

"I called ZSTD_DCtx_setMaxWindowSize(dctx,(1U<<31)); "

Yes, that's the other alternative - but it would require an update of Total Commander. Updating the limit in the dll was much faster.

[lelik007]

2ghisler(Author)
Thank you again Christian for solving the problem. I tested x64 dll the best I could and saw no issue.

[ghisler(Author)]

This was already tested with the new DLLs, and I have checked that the installer contains the updated DLLs.

Moderator message

Moved to fixed bugs