7zip native support

[ghisler(Author)]

2lelik007
Thanks for the details, that explains why I get the exact same time with and without-slp even when packing with 7zg.exe.
I thought that -slp would instruct 7z.dll to allocate memory in larger blocks at a time, not to lock them in memory.
Since I have 32GB RAM, I guess that -slp would not matter for default settings, e.g. 709MB for LZMA2 compression rate 9 and 2 threads. with these settings, packing 20 large TXT files always takes 32 seconds when using the same 7z.dll for 7zg.exe and the internal 7zip packer, with and without -slp.

What's strange is that the DLL function "SetLargePageMode" reports success although I never intentionally enabled large page mode on my system.
Is there a way to check whether large page mode is enabled or not?

[AntonyD]

A)https://learn.microsoft.com/lt-lt/windows/win32/api/memoryapi/nf-memoryapi-getlargepageminimum

B)whoami /priv | findstr SeLockMemoryPrivilege

C)

Code: Select all

#include <Windows.h>
#include <iostream>

// Функция для проверки наличия привилегии
bool CheckPrivilege(HANDLE hToken, LPCTSTR Privilege) {
    // Запрашиваем размер данных о привилегиях
    DWORD dwSize = 0;
    GetTokenInformation(hToken, TokenPrivileges, nullptr, 0, &dwSize);
    
    // Получаем информацию о привилегиях
    PTOKEN_PRIVILEGES pPrivileges = (PTOKEN_PRIVILEGES)malloc(dwSize);
    if (!GetTokenInformation(hToken, TokenPrivileges, pPrivileges, dwSize, &dwSize)) {
        free(pPrivileges);
        return false;
    }
    
    // Проверяем наличие указанной привилегии
    LUID luid;
    if (!LookupPrivilegeValue(nullptr, Privilege, &luid)) {
        free(pPrivileges);
        return false;
    }
    
    for (DWORD i = 0; i < pPrivileges->PrivilegeCount; i++) {
        if (pPrivileges->Privileges[i].Luid.LowPart == luid.LowPart &&
            pPrivileges->Privileges[i].Luid.HighPart == luid.HighPart) {
            // Привилегия найдена
            free(pPrivileges);
            return (pPrivileges->Privileges[i].Attributes & SE_PRIVILEGE_ENABLED) != 0;
        }
    }
    
    free(pPrivileges);
    return false;
}

int main() {
    HANDLE hToken = nullptr;
    
    // Открываем токен текущего процесса
    if (!OpenProcessToken(GetCurrentProcess(), TOKEN_QUERY, &hToken)) {
        std::cerr << "Failed to open process token." << std::endl;
        return 1;
    }
    
    // Проверяем привилегию SeLockMemoryPrivilege
    if (CheckPrivilege(hToken, SE_LOCK_MEMORY_NAME)) {
        std::cout << "SeLockMemoryPrivilege is enabled." << std::endl;
    } else {
        std::cout << "SeLockMemoryPrivilege is not enabled." << std::endl;
    }
    
    // Закрываем дескриптор токена
    CloseHandle(hToken);
    return 0;
}

[lelik007]

2ghisler(Author)

Is there a way to check whether large page mode is enabled or not?

In Windows 10 and earlier Task Manager if you pack a big file with a big dictionary for example 1 Gb dictionary 2 threads and 4 Gb file you can't see the real memory usage of a process if you just starts Task Manager, you see the memory is occupied by ... something, and here is how to see the real memory usage:

Spoiler

You can use "Task Manager" program to monitor the memory usage of any program running with 7-max:
Run "Task Manager"
Switch to "Details" tab in "Task Manager"
Look the following columns: "Working Set" and "Commit size". If you don't see these columns, right click in columns header, call "Select Columns" command, and select required columns.
You can track values in the following columns:
"Working Set (memory)": actual use of physical memory by the program. But this field doesn't include large pages allocated by 7-max.
"Commit size" : virtual memory allocated by program. This field includes large pages allocated by 7-max.
So, if you see a small value in the "Working Set (memory)" column and a big value in the "Commit size" column, then probably 7-max have successfully allocated large pages for the program. And you can expect that in this case the program will work faster.

It's true for Windows 10 and earlier versions, but the user report that in Windows 11 it's not like this. So if I see the memory occupied by 70% and initial Task Manager tab shows that 7z.exe uses just 20 MB - this is LP mode works.

What's strange is that the DLL function "SetLargePageMode" reports success although I never intentionally enabled large page mode on my system.

It does exactly what 7zG : tries to use LP and if it can't than uses the regular pages.
It reports: "I can't use LP so I use a fallback and continue the execution".

Since I have 32GB RAM, I guess that -slp would not matter for default settings, e.g. 709MB for LZMA2 compression rate 9 and 2 threads. with these settings, packing 20 large TXT files always takes 32 seconds when using the same 7z.dll for 7zg.exe and the internal 7zip packer, with and without -slp.

No way you can see it like this. Create an empty 2 Gb file like this:
fsutil file createnew D:\2Gb 2200000000
and pack it with 1Gb dictionary / 2 threads and then you'll see what -slp does.
I mean 7-zip with enabled or disabled LP mode.

[Horst.Epp]

For me, all these discussions about using the -slp switch are useless.
How many users would make the necessary adjustments with admin rights, including a required reboot.
Later on, they always run in this special mode with its intrinsic problems
without having a benefit for most packing jobs.

[lelik007]

2Horst.Epp
The internal packer from the beginning is meant for the newcomers and corporate users as you said it right without administrative privileges.
It starts here: viewtopic.php?p=464407#p464407 and I only tried to make clear what this thing is and what it does.

[AkulaBig]

For me, all these discussions about using the -slp switch are useless.

Check out my testing of -slp to see how useful this option is. Packing speed increases significantly.
viewtopic.php?t=84480

[Horst.Epp]

For me, all these discussions about using the -slp switch are useless.

Check out my testing of -slp to see how useful this option is. Packing speed increases significantly.
viewtopic.php?t=84480

I have seen your results, but these are only a benefit for certain large packing options.
For all other jobs, you have a much larger start-up time, which delays the whole process.
Also, this option needs a large RAM size, which is more than a medium configured system has (and needs).

There is no problem when the user knows all about it and adds the -slp manually.
All Christian should do is to write a detailed help entry, including a link to the Microsoft article.

[lelik007]

2ghisler(Author)
Christian, and about a memory calculator for internal TC's packer: level 5-9 shares bt4 match finder algo and you can see yourself in 7-zip that:
3 threads require the same amount of the memory as 2, 5 as 4, 7 as 6, and so on.

So memory required for the threads level 5-9:
3=2
5=4
7=6
This is not right that TC shows that level 5 with 3 threads requires more memory than with 2 and so on. LZMA2's bt4 doesn't work like this.

As for the levels 1-3 there's another match finder there: ht5 with the other requirements and I don't know much about those levels.
But as for level 5 - 9 you can see yourself.

[ghisler(Author)]

I have now converted the code used by 7zfm to Delphi, and I seem to get the same results as the 7zfm dialog. However, I can't choose even values for rate in 7zfm, so I can't say what the values would be there.

[lelik007]

2ghisler(Author)

I have now converted the code used by 7zfm to Delphi, and I seem to get the same results as the 7zfm dialog.

Thank you, I'll see in TC 11.50 beta 8.

[AkulaBig]

For all other jobs, you have a much larger start-up time, which delays the whole process.

No. I also tested small files. Packing time is shorter everywhere. That is, just enter this option and don’t touch it anymore.

[lelik007]

2ghisler(Author)
I've seen a new memory calculator, I can't check everything of course. But the values of memory amount I usually use match the corresponding values of 7-zip, so thank you.

[ghisler(Author)]

Many values seem to be 1 MB less than in 7zfm.exe, probably because I'm rounding down and 7zfm is rounding up(?).

[lelik007]

2ghisler(Author)
I don't usually use MB requirements so It's hard to say, but this:

7zfm is rounding up

is very possible, because it usually takes slightly less RAM than 7-zip calculates
and what is only caught my eye is a value: in TC it's 31 Gb and 32 Gb in 7-zip.
But nevertheless what is now it's a step above what is was there before.

[MaxX]

it usually takes slightly less RAM than 7-zip calculates

This depends on selected filters. No BCJ or Delta >> less RAM. The more options and files you use, the more RAM is needed.
You can see the difference of RAM consuming easy:
1. Pack 500k of files with 20GB total size
2. Pack one big TAR of the same 20GB total size.
The first one will need more RAM too.