Search files compressed with compact.exe in Windows 10

[LonerD]

Windows 10 has possibility to use NTFS files compression with built-in console command compact.exe and parameter /exe (with algoritms xpress4k, xpress8k, xpress16k, lzx).
Windows can't recognize compression, show overlay image on file icon in this case and decompress files with menu.
Total Commander have option to search compressed files, but it not works with this parameter.
Is there any way to search ? Plugins, options, scripts...?

[ghisler(Author)]

If I understand you correctly, you want to find NTFS-file system compressed files.

TC should show "c" character in "Attributes" column for NTFS-compressed files. You can also find them via the second tab of the search page.

[LonerD]

TC should show "c" character in "Attributes" column for NTFS-compressed files. You can also find them via the second tab of the search page.

It works with standard compression.

But my question - about NTFS-compression with parameter /exe in Windows 10, for example

Code: Select all

compact /c /a /f /i /exe:xpress16k /s:"d:\MyDirectory"

Files in Windows Explorer don't change color, and TC doesn't show directory and files in directory as "compressed".
In Windows file properties "Size" and "Size on disk" are diferent.
TC command cm_GetFileSpace show incorrect "Actual space used on source drive".
But maybe some tricks exist...

[Stefan2]

...
But my question - about NTFS-compression with parameter /exe in Windows 10, for example
...
But maybe some tricks exist...

How would you detect such files? (binary signatures, Header Hexdump, magic numbers)

See f.ex.:
http://file-extension.net/seeker/file_extension_zip
http://mark0.net/soft-tridnet-e.html



 

[MVV]

Stefan2,
As I understand, this 'brand new compression' doesn't change files itself, it only stores them in a new special compressed form that provides better compression for PE files. And it seems that it uses some other file attributes than reqular NTFS compressed files use (C attribute flag), and this is quite strange for me.

[Usher]

2Stefan2
2MVV
It's not "brand new", /exe switch is just for internal executable compression, now incorporated into the program for NTFS compression, see https://en.wikipedia.org/wiki/Executable_compression
The options meaning:
- lzx - LZ compression like old DOS LZEXE packer, see https://bellard.org/lzexe.html
- express4k, express8k, express16k - LZMA compression like UPX and other modern exe packers, 4k, 8k, 16k - is block size or alignment.

If it's full exe packer, every compacted executable (exe, dll etc.) should start with unpacking stub containing its unique signature (as @Stefan2 suggests). In this case there won't be "c" NTFS attribute, and TC should always show compressed size.
Maybe Total7zip plugin with additional codecs will show uncompressed size, when pressing Ctrl+PgDn to open archive…

I may be wrong, didn't test yet.

[MVV]

I think that it would be a bad idea to integrate an EXE file packer into file system level NTFS compression tool... According to this article, compression is done on file system level, so files are unpacked transparently and you can't detect the packer by signatures. Also, it is said that Explorer doesn't highlight LZX-compressed files, i.e. it doesn't really use regular C attribute, and it is a real question is there a way to detect such files or not...

[Usher]

I think that it would be a bad idea to integrate an EXE file packer into file system level NTFS compression tool...

But compact.exe IS exe packer when called with /exe switch. And now I'm sure that decompression module cannot be added as a stub to exe. I've forgotten that executable files are digitally signed with certificate so they cannot be changed. That's why decompression must be transparent.

According to this article, compression is done on file system level

It looks like I'm wrong about LZX algorithm - it's probably LZMA compression of the whole file, while express compression works on the fly with smaller LZMA compressed data blocks. Unfortunately, the article doesn't explain the whole thing.

it is a real question is there a way to detect such files or not...

If it's done on the system level, the NTFS driver must somehow recognize execompressed files so there should be some additional attribute in MFT. I know that MS coders can make it completely dumb and the NTFS driver may work like old doublspace/drivespace now checking every cluster for compression signature, but it would be really stupid. There may also be problems with defragmentation, so there should be some API dealing with new compression methods…

[MVV]

The most logical solution would using same C attribute but with some algorithm identifier within compressed sector header, but perhaps it would break backward compatibility so they did this... It is interesting what will pre-W10 system see for such compressed files. You can try compressing some EXE on a removable disk and check.

But compact.exe IS exe packer when called with /exe switch.

It is not an exe packer in usual meaning, because it has no a stub that decompresses payload into memory, it is file system driver who decompresses original file before mapping it onto the memory.

[Usher]

It is interesting what will pre-W10 system see for such compressed files. You can try compressing some EXE on a removable disk and check.

It's possible that you should first create partition and format it under newest Win10 build. It may be newer NTFS version marked somewhere there… Guessing only…

But compact.exe IS exe packer when called with /exe switch.

It is not an exe packer in usual meaning, because it has no a stub that decompresses payload into memory,

You are talking about two things - packer and unpacker. There's no need to keep both features in a single file. Remember old pkzip/pkunzip separate files?

it is file system driver who decompresses original file before mapping it onto the memory.

To be honest, I don't know (and don't want to guess any more here) whether exe pack/unpack libs are separate dlls or modules linked statically here or there. That's why I've mentioned lack of documentation.

[MVV]

Since we are talking about NTFS compression, I assume that it is a transparent lossless compression that doesn't make any file changes that are visible for applications.

You are talking about two things - packer and unpacker. There's no need to keep both features in a single file. Remember old pkzip/pkunzip separate files?

Usually EXE packers that modify binaries and leave EXE extension produce "self-extracting archives" that unpack and call compressed binaries when you execute them on the fly. Otherwise compressed files would have another extension.

[LonerD]

Is there any way to search ? Plugins, options, scripts...?

Solution.
Search with Dirsizecalc plugin
Property "Compression rate"
OP: <
Value: 100

[Usher]

Usually EXE packers that modify binaries and leave EXE extension produce "self-extracting archives" that unpack and call compressed binaries when you execute them on the fly. Otherwise compressed files would have another extension.

1. In the past there were some installers that contained packed files with extension unchanged or the archive contained only file data stream with file metadata provided separately.
2. We already know that it's not standard exepacking, so compact.exe may be only packer and some lib linked with ntfs driver may contain only exe unpacker.

It looks like corpo thinking strikes back. Maybe someone should check alternate data streams for such files?

[LonerD]

In any case TC command cm_GetFileSpace doesn't work right in current reality and essentially useless.
TC shows phrase "Actual space used", but in fact space used is different from what we see.
Image: https://i.imgur.com/2dUIXSI.png

2ghisler(Author)
Probably it's time to fix command or at least change phrase.