[regression?] copying makes fragmented files

The behaviour described in the bug report is either by design, or would be far too complex/time-consuming to be changed

Moderators: white, Hacker, petermad, Stefan2

Post Reply
wOxxOm
Junior Member
Junior Member
Posts: 12
Joined: 2004-09-25, 17:47 UTC

[regression?] copying makes fragmented files

Post by *wOxxOm »

I have a 1000GB new hard drive, no fragmentation (10% occupied), yet TC7.5 betas copy everything fragmented, however killcopy utility does not fragment (prealloc option enabled).
Options: Image: http://img233.imageshack.us/img233/7061/tc75copyoptions.png

btw, does TC use uTorrent's ultrafast method of preallocation? the one that claims new disk space without filling it with zeroes...
User avatar
Balderstrom
Power Member
Power Member
Posts: 2148
Joined: 2005-10-11, 10:10 UTC

Post by *Balderstrom »

Sparse Files
A file in which much of the data is zeros is said to contain a sparse data set.
The file compression in the NTFS file system is a partial solution to the problem. All data in the file that is not explicitly written is explicitly set to zero. File compression compacts these ranges of zeros. However, a drawback of file compression is that access time may increase due to data compression and decompression.
Utorrent: Speed up file allocation
When allocating large files uTorrent has to fill them with zeroes. That creates significant disk load and affects overall system performance. Windows XP provides a way to allocate files of any size almost instantly: SetFileValidData() function. It won't work without SE_MANAGE_VOLUME_NAME privilege (admin rights I guess) and with compressed files but I think it won't hurt anything in case of failure. Here is a link to small program that demonstrates this concept: http://depositfiles.com/files/8072561
In the latest version of utorrent, they've enabled the feature:
"diskio.no_zero"

Which from what I gather, is a workaround to the NTFS' feature that compresses large 0000 datasets.

============
Fix: of the first link, that Clo inadvertently broke w/ %40 %41's :-)
Aside: 3 email notifications for this thread, and they're all discussing my msdn link. *chuckle*
Last edited by Balderstrom on 2009-07-29, 01:01 UTC, edited 1 time in total.
User avatar
Clo
Moderator
Moderator
Posts: 5731
Joined: 2003-12-02, 19:01 UTC
Location: Bordeaux, France
Contact:

Link fixed

Post by *Clo »

2Balderstrom

[mod]I fixed the link to “Sparse files” in order to make it clickable ;)
Just needs to code the damned parenthesis beloved by M$… :P
Clo, moderator[/mod]
#31505 Traducteur Français de TC French translator Aide en Français Tutoriels Français English Tutorials
User avatar
HolgerK
Power Member
Power Member
Posts: 5406
Joined: 2006-01-26, 22:15 UTC
Location: Europe, Aachen

URL containing (VS.85)

Post by *HolgerK »

2Clo,

small correction of your fix:

http://msdn.microsoft.com/en-us/library/aa365564%28VS.85%29.aspx

BTW: simply omitting the (VS.85)

http://msdn.microsoft.com/en-us/library/aa365564.aspx

should work too.

Regards
Holger
User avatar
Clo
Moderator
Moderator
Posts: 5731
Joined: 2003-12-02, 19:01 UTC
Location: Bordeaux, France
Contact:

Kept as is

Post by *Clo »

2HolgerK

:) I know this, that occured already, but I kept the URL exactly like Balderstrom wrote it… 8)

:mrgreen: VG
Claude
Clo
#31505 Traducteur Français de TC French translator Aide en Français Tutoriels Français English Tutorials
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48092
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

TC 7.5 now uses CopyFileEx for copying by default. I have no influence on how much it fragments or not. If you turn off the new default copy method, TC will use ReadFile/WriteFile again (which has other disadvantages), but in this case TC does pre-allocate the file space before copying the actual data.
Author of Total Commander
https://www.ghisler.com
wOxxOm
Junior Member
Junior Member
Posts: 12
Joined: 2004-09-25, 17:47 UTC

Post by *wOxxOm »

:( the developers of uTorrent had also rejected the trick to preallocate files without zeroing it... however the thread on that forum shows they've changed their mind, thus speeding up preallocation and eventually eliminating fragmentation (in most cases without any performance penalty)
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48092
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

Did you try to turn off the new default copy method (which uses CopyFileEx), as I suggested? The old method did preallocate the space, but CopyFileEx doesn't have such an option.
Author of Total Commander
https://www.ghisler.com
wOxxOm
Junior Member
Junior Member
Posts: 12
Joined: 2004-09-25, 17:47 UTC

Post by *wOxxOm »

ah, now I see
Post Reply