[Implemented] Rar5 format with large dictionaries.

[lelik007]

TC 11.50b1 x64 can't unpack rar5 format file with the dictionary larger than 4 Gb, this is an empty file for example with 6Gb dictionary.
https://www.upload.ee/files/17260734/6GbEmptyFile.rar.html
It's a default limitation in UnRAR64 v7 itself, not a bug of any kind and this is for x64 only. E.Roshal did it for the safety reasons.

In Winrar v7 (a format is still rar5) dictionaries to 64 Gb and with not degree of 2 (like 5) were introduced. And for unpacking, it's about ~ 1:1, so it requires 6 Gb free RAM to unpack provided file and 64 Gb to unpack the corresponding dictionary.
Winrar itself has the special setting for this (left) and -mdx switch. And in 7-zip 24.03 (right) Igor did the same exact option and the similar switch.
[img]https://www.upload.ee/thumb/17260803/Maximum_Dictionary_Size.jpg[/img]

- 7-Zip GUI and 7-Zip File Manager can ask user permission to unpack RAR archives that
require big amount of memory, if the dictionary size in RAR archive is larger than 4 GB.
- new switch -smemx{size}g : to set allowed memory usage limit for RAR archive unpacking.

So my suggestion is to have something similar in TC where we can define the maximum dictionary for unpacking otherwise we'll not be able to unpack rar5 files with the dictionary larger than 4 Gb. We can't define this limit statically too high because we don't know how much memory a user has and how much he wants to give for unpacking.

[ghisler(Author)]

Is there a switch I can pass to unrar64 dll to unpack files with such large dictionaries? Otherwise I don't see any way to support them.

[lelik007]

In Winrar it's -mdx (default is GB) -mdx6 = -mdx6g, as for dll I'll find out later.

[lelik007]

Eugene said "UCM_LARGEDICT callback message should be processed, this is mentioned in whatsnew.txt and 2. and 3."
From Documentation:
1. Function: RARProcessFileW.htm can return error:

ERAR_LARGE_DICT Dictionary size of extracted file exceeds the dictionary size limit. This limit can be increased with MaxDictSize field of RAROpenArchiveDataEx.htm

2. In RAROpenArchiveDataEx.htm there's "Callback" parameter.

Address of user defined callback function to process UnRAR events.
Set it to NULL if you do not want to define the callback function. Callback function is required to process multivolume and encrypted archives properly.

3. And the in RARCallback.htm , this is exactly what Winrar does when it meets .rar with 6 Gb dictionary and limit of 4 Gb.

UCM_LARGEDICT
Dictionary size of extracting file exceeds the limit.
Return 1 to continue extraction or -1 to cancel.
P1 contains the file dictionary size in kilobytes.
P2 contains the current dictionary size limit in kilobytes.
To prevent the excessive memory allocation it is safer to avoid extracting too large dictionaries unless it was confirmed by user.

[ghisler(Author)]

Thanks, I didn't even check whatsnew.txt because it is dated October 2023, so I thought that he forgot to update it for the new version 7...

I'm considering to just allow the 6GB allocation when the user has enough RAM, what do you think?

[lelik007]

2ghisler(Author)
Winrar x64 v7 has the dictionaries up to 64 Gb. But It's for the developer to decide how you want to limit it.

To create an archive with 18 Gb dictionary it requires 29 Gb of RAM (32Gb PC RAM) it's not a rare case now. I have the archives with 14 GB dictionaries maximum, so my weakest PC that has 16 GB of RAM can unpack them. Of course to stay in relatively safe zone, If a person has 16 Gb RAM an archive with 14 Gb dictionary can be unpacked, 32 Gb RAM Gb - 30 Gb dictionary and so on.

[lelik007]

And if the digits seems too high - they're not. 7-zip to pack with 3072 Mb dictionary requires 32 Gb of RAM even for 2 threads and every additional 2 threads doubles the requirements. And to to pack with 3840 Mb dictionary (dictionary should cover the size of the data and 4 Gb isn't that huge amount) requires 87 Gb of RAM just for 4 threads (and it's not that many threads for now).

[MaxX]

Don't think there should be any limits (by TC itself) at all. Just let it work the same way that original DLLs can do.

[lelik007]

2MaxX
It's for Christian to decide but I've got the same opinion.

[MaxX]

I've just checked some big rar7 archives with >4GB dict.
Winrar and 7zip asks me if I want to proceed just before extraction. Can't say it is a good solution here.
May be there should be something like that in TC. Some setting for big dict's limit (default is 4 GB for both Winrar and 7zip) and an option to change this in gui settings. But this will need much more time to work on this, then test this...
It would be easier and more reliable just to remove such limits at all. Can't remember if it was any problem with RAR 4 MB dicts in the end of 16-bit era and low-RAM PCs... So why should we need this now?

[ghisler(Author)]

I have tested this with the 6Gb dictionary - when unpacking the sample file, the RAM usage increases gradually until the end of the file, when it almost reaches the 6GB. So a 6Gb dictionary doesn't need more than 6GB RAM. Therefore I will use lelik007's suggestion to automatically unpack files when the dictionary size is at least 2GB smaller than the installed RAM.

[lelik007]

Just in case 2 files with the obvious names (file sizes and dictionaries).
https://www.upload.ee/files/17273581/14Gb.rar.html
https://www.upload.ee/files/17273600/30Gb.rar.html
For 16 RAM and for 32 Gb. That's for us to check.

[lelik007]

2ghisler(Author)
Christian, Prefetch/Superfetch should free some RAM before the system swap after the execution starts and IDK about Windows XP 64 but in Vista+ it's there. I think we'll stay in the safe zone. I don't know it if matters even Winrar v6 can unpack 2 and 4 Gb dictionaries though it can't create them.

[lelik007]

2MaxX
In the official ZSTD code there's a limit of 128 Mb for a dictionary to decompress, actually Christian and Eugene had to increase it to the maximum of 1Gb for 32 bit and 2 Gb for 64 bit.

[MaxX]

For 16 RAM and for 32 Gb. That's for us to check.

Not so perfect. I have 32 G of RAM. My iGPU eats some of it. So can't get free 30 gigs even if don't use any OS at all. So I can't check second one.
I can extract 14 gig archive with total7zip plugin now, just renamed it to "14Gb.rar.7ztotsz" (a special extension for a plugin I made). But can't do this for "14Gb.rar" with unRarDll now.