Support for sparse files
Moderators: white, Hacker, petermad, Stefan2
Support for sparse files
Total Commander (7.50) ignores all sparse properties of files. When copying, it inflates them to full size, with some files this introduces tens of gigabytes of bloat. When reporting disk space, it reports the incorrect inflated size. Explorer reports files size correctly but also inflates when copying. Win32 console copy command also inflates files.
Actually the whole Windows world seems to largely ignore the NTFS file system and only Linuxes support it correctly. As a Linux-dummy I have a hard time moving to a new HD since I cannot copy sparse files from the old one without inflating them.
Linux cp command handles sparse files, it seems from the documentation.
I have not found a sparse-aware Windows file utility by googling.
It would be very nice if Total Commander were able to handle sparse files correctly.
Edited Win32 console copy command part... I think I once got it to copy correctly within one volume but I currently cannot reproduce this.
Actually the whole Windows world seems to largely ignore the NTFS file system and only Linuxes support it correctly. As a Linux-dummy I have a hard time moving to a new HD since I cannot copy sparse files from the old one without inflating them.
Linux cp command handles sparse files, it seems from the documentation.
I have not found a sparse-aware Windows file utility by googling.
It would be very nice if Total Commander were able to handle sparse files correctly.
Edited Win32 console copy command part... I think I once got it to copy correctly within one volume but I currently cannot reproduce this.
Here's some actual information on sparse files (Microsoft only provides fluff) and example source code:
http://www.flexhex.com/docs/articles/sparse-files.phtml
http://www.flexhex.com/docs/articles/sparse-files.phtml
- Balderstrom
- Power Member
- Posts: 2148
- Joined: 2005-10-11, 10:10 UTC
Archives definitely don't support sparse files. So packers will pack entire file (with zero byte areas for every missing block), and after unpacking file will be the same as in case of regular copy. Special handling is required to copy/pack sparse files (need to enum existing blocks via DeviceIoControl and read only theese blocks).Balderstrom wrote:What happens if you archive those files: zip, tar.gz, tar.bz etc. And then unpack them when they've been transferred to the new Hard Drive?
- Balderstrom
- Power Member
- Posts: 2148
- Joined: 2005-10-11, 10:10 UTC
Most of archives would pack those 0's as almost no space (even NTFS compression would - I have 1-GB file that consists of same bytes and it takes only 64 MB of physical space).Balderstrom wrote:A solid Rar would pack those 0's as almost no space.
But sparse structure would lost. Sparse file size can exceed volume size while unpacked files not.
BTW it doesn't matter is archive solid or not - solid archives are useful only when you pack multiple files with similar contents (it compresses them as solid data block).
I'm not very familiar witrh Far Manager, couldn't get it to copy sparse.
The cmdline NTFS alt. stream copy tool sparse.zip/CopyStream/cs.exe at the bottom of this page did the trick though (lightning fast compared to bloatcopy) - the original version of it, in Streamtools, does not, but this is the modified sparse-aware variant. It says "source code", but there is a binary file included.
http://www.flexhex.com/docs/articles/sparse-files.phtml
The cmdline NTFS alt. stream copy tool sparse.zip/CopyStream/cs.exe at the bottom of this page did the trick though (lightning fast compared to bloatcopy) - the original version of it, in Streamtools, does not, but this is the modified sparse-aware variant. It says "source code", but there is a binary file included.
http://www.flexhex.com/docs/articles/sparse-files.phtml
- ghisler(Author)
- Site Admin
- Posts: 48096
- Joined: 2003-02-04, 09:46 UTC
- Location: Switzerland
- Contact:
No, I switched from my own copy functions to CopyFileEx because there were just too many problems with badly written device drivers for custom hardware.
Author of Total Commander
https://www.ghisler.com
https://www.ghisler.com
Sorry friend, but I haven't the foggiest idea what "No, I switched from my own copy functions to CopyFileEx" actually means, especially in the context of this threadghisler(Author) wrote:No, I switched from my own copy functions to CopyFileEx because there were just too many problems with badly written device drivers for custom hardware.
I'm a non-programmer.
- ghisler(Author)
- Site Admin
- Posts: 48096
- Joined: 2003-02-04, 09:46 UTC
- Location: Switzerland
- Contact:
In older versions, I used a loop of ReadFile/WriteFile, so supporting sparse files would have been possible. But now I'm using CopyFileEx to copy files, and it does not have an option to copy sparse files.
Author of Total Commander
https://www.ghisler.com
https://www.ghisler.com
How about adding option "Use old copy method (ReadFile/WriteFile)", or even more general "Choose copy method", to the Copy/Move dialog?ghisler(Author) wrote:In older versions, I used a loop of ReadFile/WriteFile, so supporting sparse files would have been possible. But now I'm using CopyFileEx to copy files, and it does not have an option to copy sparse files.
That way user might use Copy/Move dialog file type filter to copy non-sparsed files with new copy method, and sparsed using old one. This, in current design, would be a 2-step operation: F5 (for non-sparsed using new copy method) - F2 (task to BTM) - F5 (old method for non-sparsed) - F2 (task to BTM), so I guess to make it a 1-step operation a good addition to copy method selector would be its own file type filter.
Donate for Ukraine to help stop Russian invasion!
Ukraine's National Bank special bank account:
UA843000010000000047330992708
Ukraine's National Bank special bank account:
UA843000010000000047330992708