Search files compressed with compact.exe in Windows 10

English support forum

Moderators: Stefan2, white, sheep, Hacker

Post Reply
User avatar
LonerD
Senior Member
Senior Member
Posts: 298
Joined: 2010-06-19, 20:18 UTC
Location: Donetsk

Search files compressed with compact.exe in Windows 10

Post by *LonerD » 2019-01-03, 17:48 UTC

Windows 10 has possibility to use NTFS files compression with built-in console command compact.exe and parameter /exe (with algoritms xpress4k, xpress8k, xpress16k, lzx).
Windows can't recognize compression, show overlay image on file icon in this case and decompress files with menu.
Total Commander have option to search compressed files, but it not works with this parameter.
Is there any way to search ? Plugins, options, scripts...?
Total Commander DreamLair - dream always, think totally, be the commander of you own life.

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 37417
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Search files compressed with compact.exe in Windows 10

Post by *ghisler(Author) » 2019-01-07, 15:18 UTC

If I understand you correctly, you want to find NTFS-file system compressed files.

TC should show "c" character in "Attributes" column for NTFS-compressed files. You can also find them via the second tab of the search page.
Author of Total Commander
http://www.ghisler.com

User avatar
LonerD
Senior Member
Senior Member
Posts: 298
Joined: 2010-06-19, 20:18 UTC
Location: Donetsk

Re: Search files compressed with compact.exe in Windows 10

Post by *LonerD » 2019-01-08, 22:25 UTC

ghisler(Author) wrote:
2019-01-07, 15:18 UTC
TC should show "c" character in "Attributes" column for NTFS-compressed files. You can also find them via the second tab of the search page.
It works with standard compression.

But my question - about NTFS-compression with parameter /exe in Windows 10, for example

Code: Select all

compact /c /a /f /i /exe:xpress16k /s:"d:\MyDirectory"
Files in Windows Explorer don't change color, and TC doesn't show directory and files in directory as "compressed".
In Windows file properties "Size" and "Size on disk" are diferent.
TC command cm_GetFileSpace show incorrect "Actual space used on source drive".
But maybe some tricks exist... :?
Total Commander DreamLair - dream always, think totally, be the commander of you own life.

User avatar
Stefan2
Power Member
Power Member
Posts: 2512
Joined: 2007-09-13, 22:20 UTC
Location: Europa

Re: Search files compressed with compact.exe in Windows 10

Post by *Stefan2 » 2019-01-09, 08:46 UTC

LonerD wrote:
2019-01-08, 22:25 UTC
...
But my question - about NTFS-compression with parameter /exe in Windows 10, for example
...
But maybe some tricks exist... :?
How would you detect such files? (binary signatures, Header Hexdump, magic numbers)

See f.ex.:
http://file-extension.net/seeker/file_extension_zip
http://mark0.net/soft-tridnet-e.html



 
Inofficial FAQs || WIKI (Deu/Eng) || TC Home (What's new? // FAQ // Download // Order // Addons // Tools // Plugins)
Erst wenn der letzte Baum gefällt oder die letzte Biene verendet ist, werden die Leute verstehen warum Umweltschutz wichtig ist.

User avatar
MVV
Power Member
Power Member
Posts: 8276
Joined: 2008-08-03, 12:51 UTC
Location: Russian Federation

Re: Search files compressed with compact.exe in Windows 10

Post by *MVV » 2019-01-10, 18:23 UTC

Stefan2,
As I understand, this 'brand new compression' doesn't change files itself, it only stores them in a new special compressed form that provides better compression for PE files. And it seems that it uses some other file attributes than reqular NTFS compressed files use (C attribute flag), and this is quite strange for me.

User avatar
Usher
Senior Member
Senior Member
Posts: 402
Joined: 2011-03-11, 10:11 UTC

Re: Search files compressed with compact.exe in Windows 10

Post by *Usher » 2019-01-11, 21:05 UTC

2Stefan2
2MVV
It's not "brand new", /exe switch is just for internal executable compression, now incorporated into the program for NTFS compression, see https://en.wikipedia.org/wiki/Executable_compression
The options meaning:
- lzx - LZ compression like old DOS LZEXE packer, see https://bellard.org/lzexe.html
- express4k, express8k, express16k - LZMA compression like UPX and other modern exe packers, 4k, 8k, 16k - is block size or alignment.

If it's full exe packer, every compacted executable (exe, dll etc.) should start with unpacking stub containing its unique signature (as @Stefan2 suggests). In this case there won't be "c" NTFS attribute, and TC should always show compressed size.
Maybe Total7zip plugin with additional codecs will show uncompressed size, when pressing Ctrl+PgDn to open archive…

I may be wrong, didn't test yet.
Regards from Poland
Andrzej P. Wozniak

User avatar
MVV
Power Member
Power Member
Posts: 8276
Joined: 2008-08-03, 12:51 UTC
Location: Russian Federation

Re: Search files compressed with compact.exe in Windows 10

Post by *MVV » 2019-01-12, 23:39 UTC

I think that it would be a bad idea to integrate an EXE file packer into file system level NTFS compression tool... According to this article, compression is done on file system level, so files are unpacked transparently and you can't detect the packer by signatures. Also, it is said that Explorer doesn't highlight LZX-compressed files, i.e. it doesn't really use regular C attribute, and it is a real question is there a way to detect such files or not...

User avatar
Usher
Senior Member
Senior Member
Posts: 402
Joined: 2011-03-11, 10:11 UTC

Re: Search files compressed with compact.exe in Windows 10

Post by *Usher » 2019-01-13, 01:43 UTC

MVV wrote:
2019-01-12, 23:39 UTC
I think that it would be a bad idea to integrate an EXE file packer into file system level NTFS compression tool...
But compact.exe IS exe packer when called with /exe switch. And now I'm sure that decompression module cannot be added as a stub to exe. I've forgotten that executable files are digitally signed with certificate so they cannot be changed. That's why decompression must be transparent.
MVV wrote:
2019-01-12, 23:39 UTC
According to this article, compression is done on file system level
It looks like I'm wrong about LZX algorithm - it's probably LZMA compression of the whole file, while express compression works on the fly with smaller LZMA compressed data blocks. Unfortunately, the article doesn't explain the whole thing.
MVV wrote:
2019-01-12, 23:39 UTC
it is a real question is there a way to detect such files or not...
If it's done on the system level, the NTFS driver must somehow recognize execompressed files so there should be some additional attribute in MFT. I know that MS coders can make it completely dumb and the NTFS driver may work like old doublspace/drivespace now checking every cluster for compression signature, but it would be really stupid. There may also be problems with defragmentation, so there should be some API dealing with new compression methods…
Regards from Poland
Andrzej P. Wozniak

User avatar
MVV
Power Member
Power Member
Posts: 8276
Joined: 2008-08-03, 12:51 UTC
Location: Russian Federation

Re: Search files compressed with compact.exe in Windows 10

Post by *MVV » 2019-01-14, 06:17 UTC

The most logical solution would using same C attribute but with some algorithm identifier within compressed sector header, but perhaps it would break backward compatibility so they did this... It is interesting what will pre-W10 system see for such compressed files. You can try compressing some EXE on a removable disk and check. :)
But compact.exe IS exe packer when called with /exe switch.
It is not an exe packer in usual meaning, because it has no a stub that decompresses payload into memory, it is file system driver who decompresses original file before mapping it onto the memory.

User avatar
Usher
Senior Member
Senior Member
Posts: 402
Joined: 2011-03-11, 10:11 UTC

Re: Search files compressed with compact.exe in Windows 10

Post by *Usher » 2019-01-14, 14:09 UTC

MVV wrote:
2019-01-14, 06:17 UTC
It is interesting what will pre-W10 system see for such compressed files. You can try compressing some EXE on a removable disk and check. :)
It's possible that you should first create partition and format it under newest Win10 build. It may be newer NTFS version marked somewhere there… Guessing only…
MVV wrote:
2019-01-14, 06:17 UTC
But compact.exe IS exe packer when called with /exe switch.
It is not an exe packer in usual meaning, because it has no a stub that decompresses payload into memory,
You are talking about two things - packer and unpacker. There's no need to keep both features in a single file. Remember old pkzip/pkunzip separate files?
MVV wrote:
2019-01-14, 06:17 UTC
it is file system driver who decompresses original file before mapping it onto the memory.
To be honest, I don't know (and don't want to guess any more here) whether exe pack/unpack libs are separate dlls or modules linked statically here or there. That's why I've mentioned lack of documentation.
Regards from Poland
Andrzej P. Wozniak

User avatar
MVV
Power Member
Power Member
Posts: 8276
Joined: 2008-08-03, 12:51 UTC
Location: Russian Federation

Re: Search files compressed with compact.exe in Windows 10

Post by *MVV » 2019-01-15, 05:49 UTC

Since we are talking about NTFS compression, I assume that it is a transparent lossless compression that doesn't make any file changes that are visible for applications.
You are talking about two things - packer and unpacker. There's no need to keep both features in a single file. Remember old pkzip/pkunzip separate files?
Usually EXE packers that modify binaries and leave EXE extension produce "self-extracting archives" that unpack and call compressed binaries when you execute them on the fly. Otherwise compressed files would have another extension.

User avatar
LonerD
Senior Member
Senior Member
Posts: 298
Joined: 2010-06-19, 20:18 UTC
Location: Donetsk

Re: Search files compressed with compact.exe in Windows 10

Post by *LonerD » 2019-01-31, 12:46 UTC

LonerD wrote:
2019-01-03, 17:48 UTC
Is there any way to search ? Plugins, options, scripts...?
Solution.
Search with Dirsizecalc plugin
Property "Compression rate"
OP: <
Value: 100
Total Commander DreamLair - dream always, think totally, be the commander of you own life.

User avatar
Usher
Senior Member
Senior Member
Posts: 402
Joined: 2011-03-11, 10:11 UTC

Re: Search files compressed with compact.exe in Windows 10

Post by *Usher » 2019-01-31, 13:18 UTC

MVV wrote:
2019-01-15, 05:49 UTC
Usually EXE packers that modify binaries and leave EXE extension produce "self-extracting archives" that unpack and call compressed binaries when you execute them on the fly. Otherwise compressed files would have another extension.
1. In the past there were some installers that contained packed files with extension unchanged or the archive contained only file data stream with file metadata provided separately.
2. We already know that it's not standard exepacking, so compact.exe may be only packer and some lib linked with ntfs driver may contain only exe unpacker.

It looks like corpo thinking strikes back. Maybe someone should check alternate data streams for such files?
Regards from Poland
Andrzej P. Wozniak

Post Reply