Support for sparse files

Here you can propose new features, make suggestions etc.

Moderators: white, Hacker, petermad, Stefan2

Post Reply
blind12
Junior Member
Junior Member
Posts: 20
Joined: 2004-04-22, 17:06 UTC

Support for sparse files

Post by *blind12 »

Total Commander (7.50) ignores all sparse properties of files. When copying, it inflates them to full size, with some files this introduces tens of gigabytes of bloat. When reporting disk space, it reports the incorrect inflated size. Explorer reports files size correctly but also inflates when copying. Win32 console copy command also inflates files.

Actually the whole Windows world seems to largely ignore the NTFS file system and only Linuxes support it correctly. As a Linux-dummy I have a hard time moving to a new HD since I cannot copy sparse files from the old one without inflating them.

Linux cp command handles sparse files, it seems from the documentation.

I have not found a sparse-aware Windows file utility by googling.

It would be very nice if Total Commander were able to handle sparse files correctly.

Edited Win32 console copy command part... I think I once got it to copy correctly within one volume but I currently cannot reproduce this.
blind12
Junior Member
Junior Member
Posts: 20
Joined: 2004-04-22, 17:06 UTC

Post by *blind12 »

Here's some actual information on sparse files (Microsoft only provides fluff) and example source code:

http://www.flexhex.com/docs/articles/sparse-files.phtml
blind12
Junior Member
Junior Member
Posts: 20
Joined: 2004-04-22, 17:06 UTC

Post by *blind12 »

Far Manager seems to have some NTFS features support, incl. sparse files... checking into it...
User avatar
Balderstrom
Power Member
Power Member
Posts: 2148
Joined: 2005-10-11, 10:10 UTC

Post by *Balderstrom »

What happens if you archive those files: zip, tar.gz, tar.bz etc. And then unpack them when they've been transferred to the new Hard Drive?
*BLINK* TC9 Added WM_COPYDATA and WM_USER queries for scripting.
User avatar
MVV
Power Member
Power Member
Posts: 8702
Joined: 2008-08-03, 12:51 UTC
Location: Russian Federation

Post by *MVV »

Balderstrom wrote:What happens if you archive those files: zip, tar.gz, tar.bz etc. And then unpack them when they've been transferred to the new Hard Drive?
Archives definitely don't support sparse files. So packers will pack entire file (with zero byte areas for every missing block), and after unpacking file will be the same as in case of regular copy. Special handling is required to copy/pack sparse files (need to enum existing blocks via DeviceIoControl and read only theese blocks).
User avatar
Balderstrom
Power Member
Power Member
Posts: 2148
Joined: 2005-10-11, 10:10 UTC

Post by *Balderstrom »

A solid Rar would pack those 0's as almost no space.
*BLINK* TC9 Added WM_COPYDATA and WM_USER queries for scripting.
User avatar
MVV
Power Member
Power Member
Posts: 8702
Joined: 2008-08-03, 12:51 UTC
Location: Russian Federation

Post by *MVV »

Balderstrom wrote:A solid Rar would pack those 0's as almost no space.
Most of archives would pack those 0's as almost no space (even NTFS compression would - I have 1-GB file that consists of same bytes and it takes only 64 MB of physical space). :)
But sparse structure would lost. Sparse file size can exceed volume size while unpacked files not.

BTW it doesn't matter is archive solid or not - solid archives are useful only when you pack multiple files with similar contents (it compresses them as solid data block).
blind12
Junior Member
Junior Member
Posts: 20
Joined: 2004-04-22, 17:06 UTC

Post by *blind12 »

I'm not very familiar witrh Far Manager, couldn't get it to copy sparse.

The cmdline NTFS alt. stream copy tool sparse.zip/CopyStream/cs.exe at the bottom of this page did the trick though (lightning fast compared to bloatcopy) - the original version of it, in Streamtools, does not, but this is the modified sparse-aware variant. It says "source code", but there is a binary file included.

http://www.flexhex.com/docs/articles/sparse-files.phtml
pk_asw
Junior Member
Junior Member
Posts: 6
Joined: 2013-10-06, 17:05 UTC

Post by *pk_asw »

It's quite pity, TC doesn't still support sparse file during e.g. copy operation :?
blind12
Junior Member
Junior Member
Posts: 20
Joined: 2004-04-22, 17:06 UTC

Post by *blind12 »

Indeed.

TC has embraced the "MS approach". Which is exactly the thing that scared people from Explorer to Windows/Total commander in the first place : )
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

No, I switched from my own copy functions to CopyFileEx because there were just too many problems with badly written device drivers for custom hardware. :(
Author of Total Commander
https://www.ghisler.com
blind12
Junior Member
Junior Member
Posts: 20
Joined: 2004-04-22, 17:06 UTC

Post by *blind12 »

ghisler(Author) wrote:No, I switched from my own copy functions to CopyFileEx because there were just too many problems with badly written device drivers for custom hardware. :(
Sorry friend, but I haven't the foggiest idea what "No, I switched from my own copy functions to CopyFileEx" actually means, especially in the context of this thread :)

I'm a non-programmer.
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

In older versions, I used a loop of ReadFile/WriteFile, so supporting sparse files would have been possible. But now I'm using CopyFileEx to copy files, and it does not have an option to copy sparse files.
Author of Total Commander
https://www.ghisler.com
User avatar
DrShark
Power Member
Power Member
Posts: 1872
Joined: 2006-11-03, 22:26 UTC
Location: Kyiv, 68/262
Contact:

Post by *DrShark »

ghisler(Author) wrote:In older versions, I used a loop of ReadFile/WriteFile, so supporting sparse files would have been possible. But now I'm using CopyFileEx to copy files, and it does not have an option to copy sparse files.
How about adding option "Use old copy method (ReadFile/WriteFile)", or even more general "Choose copy method", to the Copy/Move dialog?

That way user might use Copy/Move dialog file type filter to copy non-sparsed files with new copy method, and sparsed using old one. This, in current design, would be a 2-step operation: F5 (for non-sparsed using new copy method) - F2 (task to BTM) - F5 (old method for non-sparsed) - F2 (task to BTM), so I guess to make it a 1-step operation a good addition to copy method selector would be its own file type filter.
Donate for Ukraine to help stop Russian invasion!
Ukraine's National Bank special bank account:
UA843000010000000047330992708
Post Reply