Sparse file support

[white]

So apparently the Properties function (which is part of Windows) does something causing the duplication of the data?

It seems that way. I get the same behavior with 40GB files, also on my real system on a physical harddisk (2 TB, not an SSD drive). There must be some real data patched in the file for it to happen.

Edit:
I tested a sparse file created with fsutil and the issue did not occur. So there seems to be an issue with mksparse.exe when patching a file.

[j7n]

I heard about this feature being considered for removal immediately after it was added. The use cases for sparse files aren't very clear. I'm sure someone can use them if once they discover the feature later. It seems to be more useful in the program that makes large files for whatever reason, to reduce writes to the disk. Once these files already exist, TC can now make compacted copies of them. Right? Such as reducing a 10 gig flat file to 1 gig + 9 gig of zeros remembered. But the intial problem of writing to the disk has already been done.

[ghisler(Author)]

also on my real system on a physical harddisk

What do you mean with that? Nothing happens here on a real system when I alt+enter on a sparse file.

[white]

also on my real system on a physical harddisk

What do you mean with that? Nothing happens here on a real system when I alt+enter on a sparse file.

The thing I did was: First view the properties, then close the properties window, then search for a text in the file in TC's Lister (for example for a non-existing text).

But as said, it seems to be caused by your mksparse.exe program. When I use fsutil to make a sparse file, the issue doesn't occur. When I use mksparse.exe, the issue does occur. So it seems like your program is creating invalid files when patching files.

[ghisler(Author)]

I can reproduce the problem now also with fsutil: The problem only occurs if the sparse file isn't completely empty.

Example creation:

Type NUL > temp
FSUtil Sparse SetFlag temp
FSUtil Sparse SetRange temp 0 0x40000000
FSUtil File SetEOF temp 0x40000000
echo abcd >> temp

After Alt+Enter and searching, the available disk space is reduced by the nominal size of the sparse file, although the size hasn't changed with Alt+Enter, and the swap file hasn't grown either.

[white]

I can reproduce the problem now also with fsutil: The problem only occurs if the sparse file isn't completely empty.

Example creation:
..

Try it like this:
(Example with 40GB file, make sure you have 40GB free space or use a smaller size)
(The copy operation takes a while but doesn't need an additional 40GB space)

Code: Select all

fsutil file createnew testfile.bin 42949672956

copy con 4bytes
1234^Z

copy /b testfile.bin+4bytes

fsutil sparse setflag testfile.bin

fsutil sparse setrange testfile.bin 0 42949672956

[white]

Try it like this:

The file created like this, doesn't create the problem after viewing the file's properties. But after creating a copy of this file using TC, the copy does have the problem..

[white]

But after creating a copy of this file using TC..

Or robocopy..

The same things happens when using "copy testfile.bin nul" instead of searching with TC's Lister.

So it seems that disk space reducing after viewing properties, is not related to TC (and mksparse.exe) because it also happen when using Windows tools only.

Preliminary conclusion:

There may be a Windows bug where opening the Properties window of a sparse file with at least some allocated space, can cause incorrect reservation of disk space during subsequent reads (e.g. when running the command: copy testfile.bin nul). Selecting 2 files and viewing the properties of the 2 files together does not trigger this behavior.

The space is added to the "Total Reserved Clusters" as can be seen when issuing these commands (as Administrator):
fsutil fsinfo ntfsinfo c: (recent Windows 10 versions and up)
fsutil volume diskfree c: (Windows 11)

The space is freed when the sparse file is deleted. (You can copy the file first.)

Recommendation:

Don't view the properties of sparse files, but use other tools, for example Sysinternals' Disk Usage, if you want to know the allocated disk space. Or issue this command (as Administrator):
fsutil file layout <filename>

[ghisler(Author)]

I will disable multi-threading in blake3 for sparse files because it uses file mapping, which always causes the Windows bug. I will not disable the opening of the properties dialog for sparse files, this would be a big limitation. Since this is a Windows bug, we should try to report it to Microsoft via feedback hub. Maybe someone can reproduce the bug without Total Commander, just with Windows tools?

[white]

I will disable multi-threading in blake3 for sparse files because it uses file mapping, which always causes the Windows bug.

Confirmed.
Tested OK using TC 11.55rc6b 64-bit.
BTW, CrcBlake3BlockSize=0 (no memory mapping) works much faster on my system (tested with a normal 40GB file with mainly zeros).

Since this is a Windows bug, we should try to report it to Microsoft via feedback hub. Maybe someone can reproduce the bug without Total Commander, just with Windows tools?

I have created a batch file to reproduce the issue without using Total Commander. Use the batch file on a recent Windows 11 version. It needs a Windows version with a robocopy.exe version that supports the /sparse parameter. The batch file checks for it.

Sparse file test (run as Admin).cmd

Code: Select all

@echo off
setlocal EnableDelayedExpansion

:: Set current directory to batch file's folder
cd /d "%~dp0"

:: Settings
:: if duration the properties window is shown is too short, the issue does not always occur
set "duration=3"
:: when adjusting file size, adjust range as well (difference=4)
:: 1GB = 1073741824 bytes
set "fsize=1073741824"
set "range=1073741820"
set "validdata_length=4
set "validdata=1234"
set "filename=sparse file.bin"
set "copy_folder=copy"

:: Determine drive, folder, basename and filename for copy
for %%f in ("!filename!") do (
    set "drive=%%~df"
    set "folder=%%~dpf"
    set "basename=%%~nxf"
    set "filename=%%~ff"
    set "filename_copy=!folder!!copy_folder!\!basename!"
)

:: Check if robocopy has sparse support
robocopy /? | find /i "/sparse" >nul
if %errorlevel% NEQ 0 (
    echo ERROR: This test program needs robocopy.exe that supports the /sparse option.
    pause
    endlocal
    exit /b
)

cls
echo Sparse file test
echo:
echo Test if opening the properties window of a sparse file results in decrease of free disk space.
echo This batch file must be run as Administrator or the necessary information cannot be displayed.
echo:
echo This test does the following:
echo     1 Show total reserved clusters for the drive we are testing on
echo     2 Create a sparse file
echo     3 Copy the sparse file (sparse state will be retained)
echo     4 Show properties window of the copied sparse file (closes after a short time)
echo     5 Completely read the copied sparse file
echo     6 Show the drive's total reserved clusters again (check if it changed)
echo     7 Show information about the sparse files
echo     8 Delete the sparse files
echo     9 Show the drive's total reserved clusters again (check if it is back to normal)
echo:
echo The following settings will be used for testing:
echo     Sparse file        : "!filename!"
echo     Copied sparse file : "!filename_copy!"
echo     File size          : !fsize! bytes (but small allocated size)
echo     Show properties    : !duration! seconds
echo:
echo Existing files (with mentioned filenames) will be overwritten without warning.
pause

:: Show total reserved clusters
echo:
echo:
echo Drive !drive!
fsutil fsinfo ntfsinfo !drive! | find /i "Reserved Clusters"
echo:

:: Create sparse file
>"!filename!" <nul set /p "=!validdata!"
fsutil sparse setflag "!filename!"
fsutil sparse setrange "!filename!" !validdata_length! !range!
fsutil file seteof "!filename!" !fsize! >nul
<nul set /p "=(Sparse file created)"

:: Copy file while retaining sparse state
robocopy "!folder!\" "!folder!!copy_folder!" "!basename!" /sparse:y /njh /njs /nfl /ndl /np /ns /nc
echo (Sparse file copied)

:: Show properties for x seconds
call :show_properties filename_copy duration
echo (Properties window shown)

:: Read entire copied sparse file
copy /b "!filename_copy!" nul >nul
echo (Sparse file has been read completely)

:: Show total reserved clusters
echo:
echo Drive !drive!
fsutil fsinfo ntfsinfo !drive! | find /i "Reserved Clusters"

echo (Did the number of reserved clusters erroneously increase?)
pause

:: Show size and allocated size of sparse files
echo:
echo Sparse file information:
echo:
echo     Filename            : "!filename!"
fsutil file layout "!filename!" | findstr /i /r /c:"  Size.*)"
fsutil file layout "!filename!" | findstr /i /r /c:"Allocated Size.*)"
echo:
echo     Filename            : "!filename_copy!"
fsutil file layout "!filename_copy!" | findstr /i /r /c:"  Size.*)"
fsutil file layout "!filename_copy!" | findstr /i /r /c:"Allocated Size.*)"

:: Delete sparse files and remove copy folder if empty
del "!filename_copy!"
del "!filename!"
rd "!folder!!copy_folder!" 2>nul
echo:
echo (Sparse files deleted)

:: Show total reserved clusters
echo:
echo Drive !drive!
fsutil fsinfo ntfsinfo !drive! | find /i "Reserved Clusters"

echo (The number of reserved clusters should be normal again)
pause
endlocal
exit /b


:show_properties
setlocal EnableDelayedExpansion
set "seconds=!%2!"
for %%f in ("!%1!") do (
    set "folder=%%~dpf"
    set "filename=%%~nxf"
)
:: Validate folder exists
if not exist "!folder!" (
    echo Folder not found
    goto :done_show_properties
)
set ps1_show_properties= ^
{ ^
    $shell = New-Object -ComObject Shell.Application; ^
    $folder = $shell.NameSpace('!folder!'); ^
    if ($folder) { ^
        $file = $folder.ParseName('!filename!'); ^
        if ($file) { ^
            $file.InvokeVerb('properties'); ^
            Start-Sleep -Seconds !seconds! ^
        } else { ^
            Write-Host 'File not found' ^
        } ^
    } ^
}
powershell -Command "& !ps1_show_properties!"
:done_show_properties
endlocal
exit /b

Here is the output on screen when I run ""d:\sparse test\Sparse file test (run as Admin).cmd" as Administrator on my system:

Code: Select all

Sparse file test

Test if opening the properties window of a sparse file results in decrease of free disk space.
This batch file must be run as Administrator or the necessary information cannot be displayed.

This test does the following:
    1 Show total reserved clusters for the drive we are testing on
    2 Create a sparse file
    3 Copy the sparse file (sparse state will be retained)
    4 Show properties window of the copied sparse file (closes after a short time)
    5 Completely read the copied sparse file
    6 Show the drive's total reserved clusters again (check if it changed)
    7 Show information about the sparse files
    8 Delete the sparse files
    9 Show the drive's total reserved clusters again (check if it is back to normal)

The following settings will be used for testing:
    Sparse file        : "D:\sparse test\sparse file.bin"
    Copied sparse file : "D:\sparse test\copy\sparse file.bin"
    File size          : 1073741824 bytes (but small allocated size)
    Show properties    : 3 seconds

Existing files (with mentioned filenames) will be overwritten without warning.
Press any key to continue . . .


Drive D:
Total Reserved Clusters :                  1.024  (4,0 MB)

(Sparse file created)
(Sparse file copied)
(Properties window shown)
(Sparse file has been read completely)

Drive D:
Total Reserved Clusters :                264.175  (1,0 GB)
(Did the number of reserved clusters erroneously increase?)
Press any key to continue . . .

Sparse file information:

    Filename            : "D:\sparse test\sparse file.bin"
    Size                : 1.073.741.824 ( 1,0 GB)
    Allocated Size      :        65.536 (64,0 KB)

    Filename            : "D:\sparse test\copy\sparse file.bin"
    Size                : 1.073.741.824 ( 1,0 GB)
    Allocated Size      :        65.536 (64,0 KB)

(Sparse files deleted)

Drive D:
Total Reserved Clusters :                  1.024  (4,0 MB)
(The number of reserved clusters should be normal again)
Press any key to continue . . .

[ghisler(Author)]

Thanks for the script, I can reproduce the problem with it (I had to change "Reserved Clusters" to the localized version, though).
Allocating these clusters doesn't seem to be a bug, but not releasing them probably is. They are described as follows (e.g. here):

Reserved clusters are clusters that NTFS reserves just in case it needs to allocate space for a critical operation (like expanding a compressed file or extending the $MFT).

So NTFS sees some type of access to that sparse file, decides that it may be necessary to actually allocate the full data, and then reserves the memory for it.

On my system, even xcopy seems to trigger the problem:

Type NUL > temp
FSUtil Sparse SetFlag temp
FSUtil Sparse SetRange temp 0 0x40000000
FSUtil File SetEOF temp 0x40000000
echo abcd >> temp
xcopy /-I temp.txt temp2.txt

After the xcopy command, the size of the reserved clusters has jumped by the size of the sparse file, no "Properties" dialog necessary. Can you reproduce that?

BTW, CrcBlake3BlockSize=0 (no memory mapping) works much faster on my system (tested with a normal 40GB file with mainly zeros).

Are you sure this isn't a cache problem? Here the multi-threaded function is considerably faster on an NVMe SSD, and equally fast on a SATA SSD, and not used on a harddisk.

[white]

Thanks for the script, I can reproduce the problem with it (I had to change "Reserved Clusters" to the localized version, though).

Lucky for me, there is only the English fsutil.exe.mui file on my system

Allocating these clusters doesn't seem to be a bug, but not releasing them probably is. They are described as follows (e.g. here):

Reserved clusters are clusters that NTFS reserves just in case it needs to allocate space for a critical operation (like expanding a compressed file or extending the $MFT).

So NTFS sees some type of access to that sparse file, decides that it may be necessary to actually allocate the full data, and then reserves the memory for it.

Perhaps. But sparse files are not compressed files. Nothing needs to be unpacked. The whole point of sparse files is that you don't need to allocate any space for zero filled ranges.

On my system, even xcopy seems to trigger the problem:

Type NUL > temp
FSUtil Sparse SetFlag temp
FSUtil Sparse SetRange temp 0 0x40000000
FSUtil File SetEOF temp 0x40000000
echo abcd >> temp
xcopy /-I temp.txt temp2.txt

After the xcopy command, the size of the reserved clusters has jumped by the size of the sparse file, no "Properties" dialog necessary. Can you reproduce that?

If I fix your error in the filename ("temp.txt" should be "temp"), then yes, I can reproduce that. But you are taking a step back. It doesn't depend on xcopy. Any reading of the file will do. Like "copy temp nul" or searching for a text in the file using TC's Lister. It was already established that it matters how you create the sparse file. Perhaps using "echo abcd >>" on sparse files is not properly supported. Similarly, your mksparse tool when used to patch data, does perhaps not handle sparse files well. Since I couldn't rule this out, I created sparse files in a different way. For example, the issue does not occur when re-ordering the lines of the code you gave like this:

Code: Select all

Type NUL > temp
FSUtil File SetEOF temp 0x40000000
echo abcd >> temp
FSUtil Sparse SetFlag temp
FSUtil Sparse SetRange temp 0 0x40000000
xcopy /-I temp temp2

My batch file also creates a sparse file that doesn't have the issue of clusters being reserved when the file is being read. Even if looking at the properties of the file first. You can create multiple files like this and the issue does not occur. But when you copy the file using TC or robocopy, then it does matter if you check the properties of the sparse file. To make matters even more complex, even checking the properties of the original sparse file can trigger reserving clusters for the copied file when the copied file is subsequently being read (this only happens on my real system, not in Windows Sandbox).

BTW, CrcBlake3BlockSize=0 (no memory mapping) works much faster on my system (tested with a normal 40GB file with mainly zeros).

Are you sure this isn't a cache problem? Here the multi-threaded function is considerably faster on an NVMe SSD, and equally fast on a SATA SSD, and not used on a harddisk.

I use a NVMe SSD drive. I don't think caching is the problem. But I will do more testing later if I don't forget and I'll report my findings in a separate thread.

[ghisler(Author)]

The difference seems to be the call to

Code: Select all

FSUtil Sparse SetFlag temp
FSUtil Sparse SetRange temp 0 0x40000000

AFTER modifying the data. If I move that up, the problem occurs although the file still has the sparse attribute.

If you run the batch file step by step, the file has allocated one GB after the call echo abcd >> temp, and is only reduced to 64k after the two FSUtil Sparse calls. That's not optimal, you couldn't copy a sparse file this way if the free space is less than the nominal size of the sparse file. Your code fails if the free space is smaller than the nominal size of the sparse file.

After copying the file with CopyFile2 (F5 in TC) or with my own code to copy the sparse file (F5 with verify on), I get the error after Alt+Enter only.
I have tried calling

Code: Select all

DeviceIoControl(handle, FSCTL_SET_SPARSE,...

again immediately after copying, but it didn't help.

Btw, the following code also produces a problematic sparse file:

Code: Select all

Type NUL > temp
FSUtil Sparse SetFlag temp
FSUtil Sparse SetRange temp 0 0x40000000
FSUtil File SetEOF temp 0x40000000
echo abcd >> temp
FSUtil Sparse SetFlag temp
FSUtil Sparse SetRange temp 0 0x40000000

and this one too:

Code: Select all

Type NUL > temp
FSUtil Sparse SetFlag temp
FSUtil File SetEOF temp 0x40000000
echo abcd >> temp
FSUtil Sparse SetRange temp 0 0x40000000

So is there no way to create a sparse file except for creating the full size file an then calling SetRange for each zero part?

[ghisler(Author)]

I use a NVMe SSD drive. I don't think caching is the problem.

Which processor (CPU) do you have?
How much RAM?
I could disable multi-threaded blake3 depending on the system specifications.
Edit: I have disabled multi-threaded blake3 in RC7 now when a system has less than 8 virtual cores (real cores+hyperthreading).

[white]

So is there no way to create a sparse file except for creating the full size file an then calling SetRange for each zero part?

Yes, there is a way. But it doesn't help you for Total Commander. I used it in my batch file to illustrate the issue. Instead of adding valid data to the end, I made sparse file with valid data at the beginning by first creating a small normal file. This is the code I used:

Code: Select all

>"!filename!" <nul set /p "=!validdata!"
fsutil sparse setflag "!filename!"
fsutil sparse setrange "!filename!" !validdata_length! !range!
fsutil file seteof "!filename!" !fsize! >nul

The first line creates a small normal file with the contents of the variable validdata. I used "set /p" to avoid that a newline is added as well.
Then the sparse flag is set.
Then the sparse range is set beginning after the valid data.
Then the file is resized to the new size with "fsutil file seteof". (size = length valid data + length range)