Verify after copy

[knnknn]

You would refuse to use a server that can crash because of a power failure?

Definitely yes, if this will happen more than twice.

Everyone should read HolgerK's answer. It's too ridiculous to be believed: He wouldn't copy anything to a computer that isn't connected to an UPS.

A solid system has several mechanism to guarantee the data integrity:
- Memory with parity.
- CRC checks on physical layers.
- Checksums on logical layers.
- RAID 1/5/6 hard disk mirroring/parity
- Backups

You, see you are talking theories, while I am talking practice. I copy several terabytes over LAN every month AND COPY ERRORS HAPPEN.

How in the world can anyone be AGAINST VERIFYING?

How do _YOU_ know that a copy error never happened on your systems, since you are against verifying?

And, no, Backups do not guarantee data integrity, since Backups without Copy+Verify are the very problem I am talking about.

Faulty memory will not warn you when copy errors happen.

Faulty RAIDs will also not warn you.

And, no, CRC checks won't protect from read errors.

And how do you know the new harddrive is working OK, if you don't copy+verify? You will notice after weeks/months that x% of your files have been corrupted. ANd maybe you will never notice if you MOVED your files without verifying.

Seriously, you are talking about a imho in most cases unnecessary feature for a single program

Well, if _YOU_ don't need it then don't verify. You are the only one who should care about YOUR data. I and many other will surely welcome a verify function for the extra safety that it gives.

, while the rest of the programs (including the OS) running on such scrappy hardware, is creating one erroneous file after the another.

I didn't write anything about "one after the other". Copy errors happen not often, but from time to time. THAT is EXACTLY the problem: That they DON'T HAPPEN OFTEN. Thus if you don't verify you will not notice maybe for YEARS.

I have a whole pile of hard drives that a broken. Some break over night, some break over time. A copy+verify function is a way to notice when a hard drive starts to fail or when memory (RAM) starts to fail.

Creating & copying & verifying checksums on demand, to check your storage or network connection is one way, but without any consequence after detecting a possible source

Who says anything about "without any consequence"?

and replacing this faulty hardware

Unfortunately you can only replace the faulty hardware if you copied+verified the data onto a new harddrive. TC already has a rudimentary copy+verify (in the form of synchronize folders) but synchronizing doesn't always work and IS ALWAYS FAR MORE COMPLICATED than checking a simple checkbox.

[HolgerK]

I didn't say that i use computer only with UPS, but i surely would do so if the power line here would be such unstable, like it seems to be at your place.

Sigh, believe what you want, also that a verify function integrated in TC's copy operation will detect a harddisk failure before it comes to a data loss (never heard about S.M.A.R.T?).
My support for your suggestion is still neutral.

No further comments.
Holger

[Balderstrom]

HDD Health v3.3, freeware app that monitors HD via S.M.A.R.T.

[HolgerK]

CrystalDiskInfo 2.7.4 also freeware. Works fine with Vista.

[smok3]

and why not just generate that md5 file?
http://en.wikipedia.org/wiki/MD5#Applications
(monitoring HD health cannot guarantee that the file is correctly arrived)

[JohnFredC]

Some of us make our living at customer sites and have no control over the hardware environment. I have had an hand-full of occasions over the years where a file Move over a corporate LAN failed in some way (corrupted file, missing file, etc.), but, for whatever reasons, the failure was not reported by TC, only discovered per misadventure sometime afterwards.

I am not saying that TC was at fault in these situations, but that there was nothing I as a consultant could do about the hardware environment in which I was working ("refuse" to use it? Give me a break!).

The lesson I learned the hard way is to never use the Move command between disks or to network shares. IMHO there is insufficient verification in the TC copy process for me to blindly trust that the Copy portion of a Move has succeeded.

[knnknn]

Sigh, believe what you want, also that a verify function integrated in TC's copy operation will detect a harddisk failure before it comes to a data loss (never heard about S.M.A.R.T?).

As Google reported (since they have gazillions of hard disks) there is no real indication of a disk failure and SMART. Read errors and write errors HAPPEN constantly (= up to dozens of times in every second a drive is used). You will not be able to catch any bad file transfer with SMART. Not possible.

Moreover: Should I buy additional software for every PC/Harddrive I want to monitor? Shall I install software on a client's PCs? Moreover I have yet to find a SMART monitoring software that would see through all controlers+raid combinations. Not existing.

Moreover: SMART merely monitors hard drives, not LAN and not memory.

There is nothing that could circumvent the need for copy+VERIFY. Nothing.

[knnknn]

Some of us make our living at customer sites and have no control over the hardware environment. I have had an hand-full of occasions over the years where a file Move over a corporate LAN failed in some way (corrupted file, missing file, etc.), but, for whatever reasons, the failure was not reported by TC, only discovered per misadventure sometime afterwards.

Exactly. Exactly!

I am not saying that TC was at fault in these situations, but that there was nothing I as a consultant could do about the hardware environment in which I was working ("refuse" to use it? Give me a break!).

Exactly.

The lesson I learned the hard way is to never use the Move command between disks or to network shares. IMHO there is insufficient verification in the TC copy process for me to blindly trust that the Copy portion of a Move has succeeded.

I use FastCopy. Unfortunately FastCopy sucks at some features (for example it cannot skip existing files when moving, has no retry, verify is only via CRC, has queue but messes around with the correct sequence).

[fenix_productions]

2knnknn
How do you see that feature implemented? Really, how?

You had mentioned byte by byte comparison but you've also said that there is no way to prevent data loss and that "errors HAPPEN constantly". I've tried to not to get into this discussion but for me it seems that even if TC could do some magic to prevent Your data loss, something bad would happen despite that. Even having 100% reliable moving some airplane could fall down on wires

Someone mentioned copying files and using "Synchronise dirs" tool what IMHO is good enough but you were against and still want to move files around. You are always writing about "copy+verify" but don't want to acknowledge that Sync tool san be used for verifying.

I can't also believe that you are using branch view to select many files and copy them to one location. I've tried that and got dozen "overwrite confirmation" dialogues. Not the best way to go.

I don't think such functionality could be really useful in daily basis. In most of the cases it would just give OK. For rare occasions (like yours) I believe there are better tools for this purpose only somewhere.

You've mentioned FastCopy. Did you ask its author to add some features from your wish list? have you tried other tools (maybe TotalCopy, I don't know)?

[Balderstrom]

Theres a fairly easy solution that uses SyncTool.

Install Hardlink Shell Extension

Open DirBranch, select files in question
right click, Pick Link Source
Go to a TempFolder you've Created, for temporary Hardlinks.
right click, Hardlink Clone Here

SyncTool those files to whereever you like.

Delete the Hardlink Clones, after synchronization.

[smok3]

1. select the files, clicky on files/create crc checksums
2. copy the files to the new location
3. copy the md5 file there as well
4. double-clicky the remote md5
(at least that is what i do)

[eugensyl]

Is there any reason why Total Commander doesn't have an option to verify the files after copying/moving them?

FastCopy and all the other substitutes suck or are buggy.

Use TeraCopy integrated into TC.

[hlloyge]

I had to hop in into this discussion. As I am working in a rather big company (over 4000 employees, own mail, usenet web, backup, storage and application servers), I do much of a copying around.
Ive never, ever, lost a file during copying it over network. In those rare occasions I would get file copy error of some sort (mostly failed to write MFT on destination disk), and on destination computer/server I would check out what is going on - sometimes it was bad hard drive, or a bad network cable, of broken switch.
Windows (and other OSes) have the means of reporting if a file copy failed. If that happens, it is best to adress the problem at once.
Once the file is copied, you can do few things: regular backups help a lot. If there is no backup plan, archive it with some redundancy enabled, or create parity files and store them elsewhere.
To have file copied without any errors and then checked and to be wrong - I find that hard to believe. If that happens, there is a problem with a hardware, not the programs itself.
And I stand behind what I am telling you - I've been administering systems for ten years now, and I have computer experience since commodore times, and I understand how these things work. TCP protocol won't allow the file to loose it's bits while travelling through network, OS won't allow those bits to change while transfering them form network card to disk, HDD firmware will check if they are written right. Errors can happen then, but they will be sent over to the source, and you will see them. Any degradation will happen, if you're unlucky, because of a hardware failure one second after you're finished copying (and checking), or 10 years after, or never.

[Fuzbolero]

I think that we are not discussing a feature that will remove all potential problems with reliability here, but one that will help us save time and provide "some extra peace of mind".

As many of us are working on several computers, both our own and with clients, it is an important aspect to have such a function built-in to our main tool, without having to bother with a series of tools.

This is also relevant to dealing with normal computers, at home, etc. Not only in server/professional environments.

It is basically just about a tick mark that I believe no-one can argue would create more problems that it solves? (Again; it does not solve all problems, just gives a convenient extra peace of mind.)

[MVV]

I think here will be good another helper DLL, which will have a function BOOL CompareFiles(LPCTSTR name1, LPCTSTR name2), which will be called after each file copy operation. So, if user don't need verifying and this DLL, TC won't call this function.