Copy to usb corrupts files/filesystem

[bylie]

Hi,

I've been happily using Total Commander since 1 year now but have recently found something disturbing.

Last week I plugged in my external usb drive to copy over some files (about 5GB), after copying them over with TC I safely removed the hardware as usual. When I plugged this drive into my other computer I found that some files and directories were corrupted, a chkdsk confirmed this. Thnking : sh*t happens, I checked the disk for errors using the vendor (samsung) tools. A short and long test stated the entire disk was fine. So still not thinking much of it I reformatted the disk and copied the data back over which I also had stored on another PC, this copy was done with the normal windows explorer, after running a verify tool all data was copied over as it should.

Now I thought that was it, something fluked along the way, better be careful next time...

But today I plugged in two external usb drives into my laptop (yet another PC than those used above) to copy some data from one drive to the other. I started the copy with TC but after about 10% TC said it couldn't read anymore data so I had to abort. After trying a couple times more the entire ntfs on the source disk is borked and isn't recognized anymore by windows. So again I did a check with the vendor (seagate) tools and this turned out OK. I haven't had a look on the target drive but as it was empty to begin with it wasn't of much interest to me.

So I'm beginning to suspect TC for the corruption happening, luckily most data was not that important to me and can be missed (some movies etc) and the important data I still had an offline backup laying around.

I looked around in the configuration settings and in the copy/delete tab I (for now) set everything to windows explorer instead of TC's build in copy/delete.

I'll be away until friday (02/05) when I plan to do some more tests on this subject to verify some things.

Specs :
Total Commander 7.0.2a
Windows XP SP2

[karlchen]

Hello, bylie.

There are quite a few threads where similar problems have been reported. As a matter of fact, configuring TotalCommander to use compatibility mode on the affected drives (or on all drives) should solve the problem as well.
In compatibility mode you can still use the descript.ion functionality whereas in Windows Explorer mode descript.ion files will be disregarded.

Regards,
Karl

[DrShark]

Try this plugin:
http://forum.wincmd.ru/viewtopic.php?p=40638#40638 (Translated description)
It will not recover already broken files, but it will help to protect data from damage in the future.

[ghisler(Author)]

I doubt that Total Commander is to blame here:
1. Total Commander uses the most simple copy method possible, a loop of ReadFile and WriteFile. If detects an USB stick (which must NOT be formatted as a harddisk), then TC will use compatibility mode automatically!
2. USB drives use a standard driver which is part of Windows, and which does NOT have any problems with WriteFile. The write problems always had to do with custom RAID controllers or special network drives (NAT)
3. Because of the heavy price war there is a big danger of getting a bad or even fake USB sticks reporting a larger size than they can really hold.

I still recommend that you try to switch to compatibility mode manually and make some copy tests: Configuration - Options - Copy/Delete. If some files still get damaged, then the stick is definitely damaged. Otherwise the stick controller may not be able to handle WriteFile of arbitrary block sizes, which means that you would also get data loss when saving documents directly to the stick...

There are quite a few threads where similar problems have been reported.

With USB sticks? Where?

[karlchen]

Hello, bylie.

If you wish to follow Christian's advice and make sure that your USB stick is all right, I recommend to download H2Testw v1.4 by the German computer magazine c't. Do not worry, the little programme does speak English as well.

H2Testw has been created for the sole purpose of verifying that a USB stick works as expected and that it has got the capacity which it should have. Quite a few faked sticks have been sold pretending to be 2GB-USB-sticks whereas in fact they were only 1GB-sticks. As a result at minimum the 2nd GB of data was sent to /dev/null (nowhere) or even worse overwrote the 1st GB of data.

Anyway, I have used H2Testw on my own 4GB USB stick and now know it is a 4 GB USB stick and works fine.

HTH,
Karl

[JohnFredC]

I would also point out that some USB 2.0 sticks and adapters (for SD cards, etc) are NOT backward compatible with USB 1.1 or 1.0.

For instance, yesterday I bought an inexpensive generic 2Gb USB 2.0 stick. Writes to it eventually failed on three different laptops (older Dells and a Toshiba) whose ports are 1.1 compatible (only). It didn't matter what tool was used (TC, Explorer, etc) or that I used compatibility mode in TC.

I've unfortunately had this same experience several times in the past with various so-called USB 2.0 devices.

[bylie]

Maybe a bit of confusion but the USB drives I was talking about were not USB sticks but actual harddrives in external USB enclosures. I'll try to do some more extensive tests on friday to pinpoint what the problem might be.

Edit : I actually have copied data to USB sticks with TC a number of times without any problems.

[karlchen]

Maybe a bit of confusion but the USB drives I was talking about were not USB sticks but actual harddrives in external USB enclosures.

It occurred to me only after I had posted the link to H2Testw that you might be talking about a USB harddisk really.

Yet, using compatibility mode for USB harddisk drives as a rule should avoid any data corruption.

Karl

[bylie]

Yet, using compatibility mode for USB harddisk drives as a rule should avoid any data corruption.

Karl

What are the consequences of using compatibility mode instead of the default mode for all drives (*)?

[karlchen]

To the best of my knowledge, the only downside of compatibility mode is that you can no longer use the buffer size tweaking options offered in the same dialogue box.

Karl

[ghisler(Author)]

I'm not aware of any such problems with USB harddisks either. However, Total Commander doesn't use compatibility mode for these automatically, so it's worth trying.

[bylie]

As promised I did some more extensive testing today and the results surprised me actually.

Test setup :

• Total Commander 7.02a
  Windows XP SP2
  AC Ryan External 2xIDE USB duo enclosure
  AC Ryan External SATA USB enclosure

The test consisted of copying a mix of files, varying greatly in size and type, from the local harddrive to the external USB drive. Testing was done changing TC's copy/delete settings and afterwards doing a rar check and a sfv check. While I could sum up all the results I noted during testing the results are actually very clear : Everything copied to the external IDE drive was corrupted, it didn't matter if I attached another harddisk, IDE cable, USB cable, poweradapter, changed TC's copy/delete settings or did a manual "sync -r" on the CLI before safely removing it.

OK so the external IDE enclosure will be send to the trashcan, but the odd thing is the first corruption I noticed (as explained in the openingpost) was on the external SATA enclosure. I've put this one to the same tests and checks as the IDE enclosure and all of them came back clean which surprised me.

Now the only safe conclusion I can draw from this is that TC was *NOT* at fault here but the external IDE controller was. But this still doesn't explain why the data corrupted on the SATA drive the first time, the sad thing is I can't seem to reproduce this one which leaves me with a bitter taste and still not trusting the entire chain fully.

EDIT

I just tried connecting the external IDE controller to another PC I have laying around and did some basic tests and on this PC everything seems to work and checks out OK. So I immediately swithed back to the first test PC and files started getting corrupt again. Could it be that the problem could be the combination of the external IDE controller and the mainboard or maybe some windows setting that's different, write cache handling maybe?

[ghisler(Author)]

Yes, this could indeed be the case. I made some research on the net, and there are indeed such reports. For example, someone reports that an Nforce 4 chipset starts to malfunction if it reaches more than 42°Celsius temperature (both USB and Sata). So first make sure that your system is sufficiently cooled. Also make sure you use the same cable in both cases, there can be problems with cables which are too long or internally broken (causing reflections).

Finally you should try a different IDE controller (e.g. an USB stick) on both computers to verify whether the problem is within the Computer's chipset, or the incompatibility between PC chipset and USB device chipset.

[bylie]

Yes, this could indeed be the case. I made some research on the net, and there are indeed such reports. For example, someone reports that an Nforce 4 chipset starts to malfunction if it reaches more than 42°Celsius temperature (both USB and Sata). So first make sure that your system is sufficiently cooled. Also make sure you use the same cable in both cases, there can be problems with cables which are too long or internally broken (causing reflections).

Finally you should try a different IDE controller (e.g. an USB stick) on both computers to verify whether the problem is within the Computer's chipset, or the incompatibility between PC chipset and USB device chipset.

I'm also beginning to suspect the NForce 2 chipset on this motherboard. When I think of it I actually always connected the USB IDE enclosure to my intel systems which seem to work fine with it. So the NForce 2 system now has a no USB policy and because it was an aging system anyway I've decided to reinstall it with linux so it can serve as an internet PC for the rest of the family.

Maybe a little offtopic but I'm really desperate for freeNAS 0.7 with freeBSD 7.0 and ZFS support to be released so I can finally go towards a central NAS with the security that ZFS provides (disk scrubbing, hash comparing on every read/write).