Sparse file support

[ghisler(Author)]

The problem is that sparse files usually consist of multiple valid data blocks. But as soon as I append data after calling sparse setrange, the problem occurs again.

Could you please also answer my question about your system?
Which processor (CPU) do you have?
How much RAM?

I have tested with 3 systems:
- Intel Core i7-11700 (my main desktop), Win 11
- AMD Ryzen 5 Pro 4650U (my laptop), Win 11
- Intel Core i5-760 (my old PC), Win 10
The multi-threaded function was considerably faster on the first two on NVMe SSDs, and equally fast on a SATA SSD.
It was slower on the third, older system with only 4 cores.
So for now I have disabled the multi-threaded blake3 function when there are less than 8 cores.
If you provided more info about your system, I could maybe extend this?

[ghisler(Author)]

Since this problem can only be fixed by Microsoft, I consider sparse file support broken in Windows. I will therefore disable the functions to copy sparse files in Total Commander 11.55. Users will still be able to enable it via wincmd.ini [Configuration] CopySparseFiles=1, but the default will be CopySparseFiles=0 now.

[AntonyD]

FAR / FreeCommander / Multi Commander / Double Commander / Directory Opus / Unreal Commander / One Commander / FileVoyager

But at least was this list of modern and accessible managers checked for support of this technology?
It is quite possible that none of them work with it. Or perhaps the opposite - someone is working with it
and very confidently. And it makes sense to try to understand - HOW this is happening)))

[ghisler(Author)]

Why does it matter what other third party file managers do? The bug is in Windows itself, not in Total Commander. You can reproduce it entirely without Total Commander. It affects all files which were modified after being set to sparse, which will inevitably happen when a sparse file is actively used, e.g.
- virtual machine images (Android emulator uses sparse files and writes to them)
- some torrent downloads
- large databases which aren't just read only
All these sparse files will only remain sparse as long as they are not accessed randomly without checking the sparse ranges first, which defeats the whole purpose of sparse files.

[AntonyD]

I certainly do not claim to have complete knowledge and understanding of this technology. BUT shouldn't working with such files be ensured by THIRD-PARTY SOFTWARE? We are specifically not taking into account support for this in Windows itself! All these commands/utilities like fsutil/PowerShell/robocopy - yes, they work with this type of file. BUT! they are NOT the main tool for this.
ALL work should, in theory, go through Win32API commands. This includes copying/moving such files. And it must be at the level of the third-party application code! In the context of supporting work with such files, similar managers - like our Total - should only support methods for copying/moving/getting information about such files. But their creation is definitely not their strong suit. And if currently Total cannot correctly move such files - then this is a bug in Total, not in Windows.

P.S.
Well, if we are definitely sure that this is a problem within the source code of Windows itself - then since this is a Windows bug,
shouldn't we try to report it to Microsoft via the feedback hub?

[ghisler(Author)]

And if currently Total cannot correctly move such files - then this is a bug in Total, not in Windows.

Not really - I'm using two different methods to copy sparse files:
1. On recent Windows 11 versions, CopyFile2 supports sparse files.
2. On older versions, I get the list of blocks containing data and only copy these blocks.
Both methods create sparse files which exhibit the allocation problem. So if even the built-in Windows function CopyFile2 has this problem, you can hardly blame Total Commander for it.

[AntonyD]

I didn't even try to somehow restrict/blame the capabilities of the new Total's code. Not at all! Whatever is done is done only for the good.
BUT! And the idea to check ALREADY existing managers for COMPARING their work under similar conditions - doesn't seem to be something bad or inconvenient... IMHO....

[ghisler(Author)]

Well, copying sparse files is a niche application, so I'm disabling it by default for now. Those who really need it will have to enable it via wincmd.ini. Maybe I will add a checkbox to settings in a later version, but this would require translation. I don't install any competitors so I can't be accused of stealing functions. I did some research, but only found sparse file copying mentioned in the Opus forum. Apparently they also implemented it as an optional function.

[AntonyD]

For a regular user, it will be quite difficult to identify such files in principle, since with any default settings in Total or other similar programs, it is simply impossible to find/see such files. If only it were possible to already find/select such files with the help of Total! Only then could testing be conducted more calmly and comfortably. Because using only console commands for this is somehow not very pleasant or stable.
Everything tools seems to be able to search such files: https://www.voidtools.com/forum/viewtopic.php?p=68523#p68523
But not only the holders of this tool should be able to locate such files, isn't it?

Recently, there was a somewhat heated debate about the fact that the attributes column for files is displayed and used by many people.
However, I wrote that quite the opposite is true - in my experience - 'almost no one' among my acquaintances uses it. But here, this column
would probably finally come in handy - if this new (or has it been known for a long time??) file attribute were indicated with a new letter.

[AntonyD]

By the way, I tried to use the search from that utility - and I got quite strange results. If for files that were downloaded using qBitTorrent, I can still understand the fact of creating sparse areas in them, because they are indeed very huge. But what does this mean for a 1kb file? Yes, I found such files too. BUT I can't even imagine what could fit in the sparse area, considering that ALL the content in this file is text and it actually takes up this 1KB. Maybe the search in this utility is broken? That's when a completely separate tool came in handy - namely the ability to search for such files using code in Total....

[Dalai]

For a regular user, it will be quite difficult to identify such files in principle, since with any default settings in Total or other similar programs, it is simply impossible to find/see such files. If only it were possible to already find/select such files with the help of Total!

Well, I do think such files can be found. The field tc.sparse has been there for a very long time. It goes back to at least version 6.50 (over 20 years old). The Attributes plugin also has such a field.

But it doesn't change what Ghisler wrote: Sparse files are a niche use-case. And apparently Microsoft did another typical Microsoft move yet again, implementing something they saw somewhere else but left it in an inferior or even broken state. Sparse files are supported by NTFS, but unlike compressed files this feature doesn't seem to be transparent to the application. I don't see a reason to reinvent the wheel in every application just to support sparse files. IMO it's much better to let the file-system take care of such things because it already knows about which parts of a file are sparse.

[AntonyD]

Ok, sounds logically...))

[Tahattmeruh]

I have two questions.

a) Would it be possible to show that a file is sparse in the Attr column? Currently I cannot see if it's a sparse file.

b) I have read all the posts but I don't understand why it was disabled?
For me it looks like you have a sparse file which isn't really working in any way.
What's worse when you enable the feature?

[white]

a) Would it be possible to show that a file is sparse in the Attr column? Currently I cannot see if it's a sparse file.

In the second post I wrote:

• Create a custom column view in TC that includes "tc.sparse" and "tc.size".

---

---

b) I have read all the posts but I don't understand why it was disabled?
For me it looks like you have a sparse file which isn't really working in any way.
What's worse when you enable the feature?

The sparse files seem fine. But it seems that the Windows NTFS driver doesn't handle them properly.

If you enable the feature, sparse files are copied with their sparse state retained. The problem is that after copying, working with the file can cause incorrect disk space reservation. The whole point of sparse files is that you don't need to allocate space for a bunch of zeros. But space for the whole file size gets reserved, leaving you with less free drive space. The space is freed when you delete the new sparse file. You can't see any difference between the original file and the copy (including the sparse state), but something goes wrong when working with the copy.

[ghisler(Author)]

b) That's indeed the reason why I disabled it. The problem isn't copying of sparse files, the problem is modifying sparse files after they have been created (what CopyFile2 does). So even if you never copy a sparse file, the problem will occur when you modify even one byte in that sparse file.