Verify after copy

[HolgerK]

Personally, I'll need the CRC check also on the synchronization tool ...

A synchronization between the local file system and a file system presented by a file system plug in is possible.

Regards
Holger

[HolgerK]

There was NO NOTIFICATION whatsoever.

May i ask you what SMART monitoring tool you are using that does not deliver any warning?
I'm interested in getting some concrete user experience about the usefulness of SMART or other hardware monitoring tools which claimed to work but don't?

In my experience there are hardly ever read errors because a system with read errors is unstable (crashes, audio glitches etc)

Once again, it seems you are using this faulty hardware not only for copying and moving(Backup), but also in combination with another program like player or so.

Any (in my opinion false) security feeling added by a verify after copy from TC (or any other tool) is obsolete if you use only one single program that may modify data on your disk (this may be a program that changes meta data or simply the disk de-fragmentation task from your OS).

Only a dedicated additional file like a SHA, or MD5 would warn you in this case, and this warning will only happen if you check the consistency of your data before every use (also if you read the data outside TC).

Besides of this: your data will be nuked anyway in this case.

Regards
Holger

[byblo]

Personally, I'll need the CRC check also on the synchronization tool ...

A synchronization between the local file system and a file system presented by a file system plug in is possible.

What?

There was NO NOTIFICATION whatsoever.

May i ask you what SMART monitoring tool you are using that does not deliver any warning?
I'm interested in getting some concrete user experience about the usefulness of SMART or other hardware monitoring tools which claimed to work but don't?

S.M.A.R.T. is mainly about the HDD itself (Self-Monitoring, Analysis, and Reporting Technology).
I'm not sure that you can expect that it will rely every error that can happens on a file transfer process.

Also, for instance, by using an external case for HDD which use USB for files transfert, or a simple USB adaptator, will make you lose the access to the S.M.A.R.T. informations...

Once again, it seems you are using this faulty hardware not only for copying and moving(Backup), but also in combination with another program like player or so.

Any (in my opinion false) security feeling added by a verify after copy from TC (or any other tool) is obsolete if you use only one single program that may modify data on your disk (this may be a program that changes meta data or simply the disk de-fragmentation task from your OS).

Only a dedicated additional file like a SHA, or MD5 would warn you in this case, and this warning will only happen if you check the consistency of your data before every use (also if you read the data outside TC).

Besides of this: your data will be nuked anyway in this case.
Holger

Well... a lot of things can corrupt a file, viruses, badly programmed application, defective hardware, playing with matches in your room, supernovae explosion, aliens attack, whatever.

But it seems you are missing the point from this topic, which here is that we are requesting for an automated CRC check during the moving/copying files process, nothing else.

What happens to our files before and after, is another topic.
And yes, making CRC files to be able to check your files integrity, is obviously recommended.

Sorry for bad english hope its enough understandable.

[HolgerK]

What?

A file system plug-in can simply show the real file system.
Any write operation using the file system plugin should use the write routines implemented by the plugin.
And synchronize directories is supported with file system plugins, isn't it?

S.M.A.R.T. is mainly about the HD..

This was a question to knnknn who stated that a interrupted power supply of his internal HD was causing his last problem.

But it seems you are missing the point from this topic, which here is that we are requesting for an automated CRC check during the moving/copying files process, nothing else.

No i don't miss the point but I'm truly in doubt that implementing such a feature would help you in normal file operations.

The only additional profit is that you will do the task of testing the reliability of the transfer every time you copy or move files (at a noticeable performance cost).
In my opinion this is what you should do once with new or unknown hardware or in regular intervals like an inspection. And you can do it already using "Checksum calculation/verification" or "Synchronize directories : compare by content".

In normal file system operation a dedicated md5 or sha file (which may be handled automatically by TC as durable per file information just like the description comments) is something that would help more users in most cases.
E.g. if you want to check the consistency of your data after one year.

Simply CRC checking the source and destination during or directly after a copy or move operation and forgetting the result afterwards is too less (and in most cases already done by OS and drivers).

For most users nothing else than a wrong feeling like "my files are stored secure" while the statement "my file is copied secure" may already be wrong because of file system caching effects.

Regards
Holger

[SQUIRE]

@knnknn:

Is that what you called *cough* an argument?

Nope, it's an statement of fact. Can't you tell the difference? And I'd get that cough seen to soonest - it might be swine flu.

He answered that CRC calculation is impossible with the default copy method. That's all. He could never claim that CRC is impossible per se, BECAUSE OTHER FILE COPIERS ALREADY HAVE CRC checks.

So you were told that your fetish was infeasible by TC's author. Nevertheless that has not prevented you from insisting that Ghisler accommodate your demands. So based on unreliable evidence from someone's faulty PC he would have to redesign the current system. And you claim, "how would an OPTION be overkill?" ROFLMAO.

Years ago I also did not SEEM to have copy errors until I found that for months my LAN transfers had been corrupted.

Oh dear, that does sound like a faulty power supply to me. Have you had it checked?

That was just one of my examples. I encountered all kinds of copy errors for all kinds of reasons.

How unfortunate for you. Strange that no one else here has reported having their data corrupted by these evil gremlins as far as we know. Odd that MS isn't being crucified by 100s of millions of irate Explorer users who have lost their data through it also having no auto copy validation function. Could it possibly, just possibly mind you, be YOUR rubbish equipment that is at fault and not TC's shortcomings, do you think?

So now that you've replaced your PSU with a hopefully bigger and better one, are you STILL suffering from 'all kinds of copy errors for all kinds of reasons' ??? Are Teracopy and 'other file copiers' still flagging up CRC errors that bad old TC allows through unscathed? Yes? Then show Ghisler your incontrovertible proof. No? Then you don't have a logical leg to stand on, do you?

[knnknn]

I'm interested in getting some concrete user experience about the usefulness of SMART or other hardware monitoring tools which claimed to work but don't?

Google with their thousands of HDDs posted their SMART experience. SMART is basically useless.

In my experience there are hardly ever read errors because a system with read errors is unstable (crashes, audio glitches etc)

Once again, it seems you are using this faulty hardware not only for copying and moving(Backup), but also in combination with another program like player or so.

I am not using any faulty hardware. I am not using any player in combination with copying.

I simply copied a file from one harddisk to another and it arrived broken.

Thanks CRC I didn't lose data.

Any (in my opinion false) security feeling added by a verify after copy from TC

CRC doesn't give FALSE SECURITY, it gives INCREASED security because it can and DOES catch copy errors. Nobody claims that it is perfect but it saved me from data loss a lot of times.

[knnknn]

No i don't miss the point but I'm truly in doubt that implementing such a feature would help you in normal file operations.

There is nothing to DOUBT. I am using already alternative file copiers (TeraCopy, FastCopy) and they DID catch copy errors.

I am talking EXPERIENCE. All the counter-"arguments" are based on some THEORIES.

In my opinion this is what you should do once with new or unknown hardware or in regular intervals like an inspection.
And you can do it already using "Checksum calculation/verification" or "Synchronize directories : compare by content".

There is nothing to "compare by content" if you MOVED the file and now it is corrupted.

In normal file system operation a dedicated md5 or sha file (which may be handled automatically by TC as durable per file information just like the description comments) is something that would help more users in most cases.

Yes, this would be great if TC would offer such a function: Calculate the checksums and store them centrally and then optionally compare with every copy/move. But this is already a step too far. Let's first start with a simple "Verify after Copy"

[knnknn]

Nope, it's an statement of fact. Can't you tell the difference? And I'd get that cough seen to soonest - it might be swine flu.

So you were told that your fetish was infeasible by TC's author. Nevertheless that has not prevented you from insisting that Ghisler accommodate your demands. So based on unreliable evidence from someone's faulty PC he would have to redesign the current system. And you claim, "how would an OPTION be overkill?" ROFLMAO.

Oh dear, that does sound like a faulty power supply to me. Have you had it checked?

How unfortunate for you. Strange that no one else here has reported having their data corrupted by these evil gremlins as far as we know. Odd that MS isn't being crucified by 100s of millions of irate Explorer users who have lost their data through it also having no auto copy validation function. Could it possibly, just possibly mind you, be YOUR rubbish equipment that is at fault and not TC's shortcomings, do you think?

So now that you've replaced your PSU with a hopefully bigger and better one, are you STILL suffering from 'all kinds of copy errors for all kinds of reasons' ??? Are Teracopy and 'other file copiers' still flagging up CRC errors that bad old TC allows through unscathed? Yes? Then show Ghisler your incontrovertible proof. No? Then you don't have a logical leg to stand on, do you?

I just wanted to answer your post. But then I realized... I don't have to.

[Sob]

There was this post almost a year ago, that went quite unnoticed:

I think here will be good another helper DLL, which will have a function BOOL CompareFiles(LPCTSTR name1, LPCTSTR name2), which will be called after each file copy operation. So, if user don't need verifying and this DLL, TC won't call this function.

It's not perfect, it disqualifies non-programmers and won't work with plugins, but it's closest to usable solution so far.
There are simply too many ways how this verification should work. Can we trust the source? Can we trust the destination? Is it realistic to read the file twice (e.g. over slow network)? Are some pre-computed checksums present somewhere? If so, where? And can we trust them? Etc, etc, etc... This all leads to several very different solutions to the problem. I don't think TC can make everyone happy. But external DLL could do almost anything.

[Hacker]

[mod]OK guys, if I see one more insult at a personal level, this thread gets locked for one week.

Hacker (Moderator)[/mod]

[HolgerK]

Google with their thousands of HDDs posted their SMART experience. SMART is basically useless.

You are refering this?

Our analysis identifies several parameters from the drive’s self monitoring facility (SMART) that correlate highly with failures.
Despite this high correlation, we conclude that models based on SMART parameters alone are unlikely to be useful for predicting individual drive failures. Surprisingly, we found that temperature and activity levels were much less correlated with drive failures than previously reported.

You know the difference between prediction and detection?

So you don't use any hardware monitoring tool. Thanks for information.

I am not using any faulty hardware.

Hmmh:

This was a question to knnknn who stated that a interrupted power supply of his internal HD was causing his last problem.

Any (in my opinion false) security feeling added by a verify after copy from TC

... (or any other tool) is obsolete if you use only one single program that may modify data on your disk (this may be a program that changes meta data or simply the disk de-fragmentation task from your OS).

CRC doesn't give FALSE SECURITY, it gives INCREASED security because it can and DOES catch copy errors.

You should read, understand and quote my complete posting.
I was talking about a false security feeling.

For most users nothing else than a wrong feeling like "my files are stored secure" while the statement "my file is copied secure" may already be wrong because of file system caching effects.

It's a wrong feeling because the verification only states that the copy may be successful, but not that the data are the same after 1min, 1h, 1d, 1w, 1m, 1y,..

There is nothing to DOUBT. I am using already alternative file copiers (TeraCopy, FastCopy) and they DID catch copy errors.

My doubt was not about the ability of specialized tools in detecting a faulty copy, but more about the usefulness of such a feature if in normal every day use no copy error comes up, which seems to be not in your case.

Is this normal?

Regards
Holger

[SQUIRE]

@ Hacker:

Well, for that to happen two exactly opposite mistakes would have to happen. For instance in a 1 GB file if a bit is changed on a certain position during copy, then the same bit would have to be changed back during read from target...

Um, not quite so convenient unfortunately. If there were 1000 instances of the name 'Hacker' in the input from your file, the checksum will usually catch the change of one of them to 'Hucker'. However, if I were to change all 1000 of them to 'Hucker' it is entirely possible for the hash algorithm to calculate the same checksum as it would have for the unaltered data because there will mathematically always be many different data inputs that resolve to that same checksum. Hash collisions if you'd like to look at it that way. For the same reason it is actually quite trivial to alter data to fit a particular CRC if you want i.e. a CRC is no guarantee of absolute data integrity. CRCs are only good at detecting error conditions when the number of bit errors is relatively low.

That a faulty PSU (or faulty RAM, or faulty HDD, or overheating CPU, or anything else) would cause mistakes at the exactly same position at two different specific times twice is highly improbable...

I repeat, what gives you the confidence to make that assertion, especially with a fault that can affect every single subsystem in that machine at any time to any degree of severity?

Assuming we are talking about creating a CRC from the target data, which we assume is a bad copy of the original data, then we either get correct CRC from bad data, thus not matching the CRC from good data, or we get wrong CRC from bad data, again, not matching the CRC from the good data.

Please remember that you are on an intermittently failing PC. You could never point to any data on that machine and assume anything, least of all that this is 'good' and that is 'bad' data. All data has now to be treated with suspicion because both your source and target drives are at the mercy of the same flaky PSU. That much is bleeding obvious in knnknn's plaintive cry:

And usually there are no CRC errors when you copy from (S)ATA-harddisk to (S)ATA-harddisk WITHIN THE SAME PC.
But this was the first time that a whole file became COMPLETELY unusable. It was ESPECIALLY strange since it was within the same PC.

And I might add, one cannot even dare claim that this particular CRC is 'correct' and that one is 'incorrect' whichever software program created it, be it TC or Teracopy, because you're calculating and comparing them on a potentially shaky platform. Until the problem component is replaced, you really cannot know if anything you're getting is valid or not. Is it somehow still not clear that anyone putting his faith in a validation tool on this machine under these conditions is living in a fool's paradise?

[Hacker]

SQUIRE,

if I were to change all 1000 of them to 'Hucker' it is entirely possible for the hash algorithm to calculate the same checksum as it would have for the unaltered data because there will mathematically always be many different data inputs that resolve to that same checksum. Hash collisions if you'd like to look at it that way.

But, you are dismissing hash functions as a means to verify data integrity entirely. I think any good hashing function can be used for the purpose of verifying data integrity. (SHA256 > MD5 > CRC32)

what gives you the confidence to make that assertion, especially with a fault that can affect every single subsystem in that machine at any time to any degree of severity?

Exactly that - uncertainty. You are saying both that there are a million possible outcomes and that it is probable that the outcomes will be identical. I am saying a failing system will give lots of errors (for instance incorrect checksums) rather than appear completely normal.

You could never point to any data on that machine and assume anything, least of all that this is 'good' and that is 'bad' data.

Yes, but that absolutely does not matter. The question is - do the two data sets equal on a failing system or do they not?

one cannot even dare claim that this particular CRC is 'correct' and that one is 'incorrect' whichever software program created it, be it TC or Teracopy, because you're calculating and comparing them on a potentially shaky platform.

I guess I really do not understand. Are you saying that on a failing system, with data highly susceptible to corruption, everything would start failing except for the hashing functions, which would always show that the data matches the hash?

Roman

[Sob]

2SQUIRE:
Just one thing - you're assuming that someone knows about his failing hardware and wants to "save it" by using CRCs? No doubt it's not the way to go in the long run.
But if you have seemingly correctly functioning system, how do you find out about it actually slowly failing? I mean some small errors. Bad memory for example. You can have lets say 2GB of which just one bit is bad and always returns zero. It happens and it can go unnoticed for years. Programs won't be crashing (at least not often), because it's not very likely that some important part of code ends up at this exact location in memory. More likely it'll be used by disk cache and slowly corrupting the files. And will you notice one changed byte in e.g. some big video file? Hardly, most likely it'll play just fine. Paranoid CRC checking would help you to find the problem much sooner.

[SQUIRE]

@ Sob:

Always on/off bit on an otherwise correctly functioning system? Your POST will usually immediately catch it on boot up. This sort of fundamental memory test dates back to the dawn of computing. Your memory controller also parity-checks automatically. So does your CPU. So does the HDD controller that decodes the written wave pattern back to a 1 or 0. So does the serial I/O chip that sends/receives data at the other end of your SATA cable....

'Going on for years' is not likely. If that were ever to be true unlucky businesses could be bankrupted in seconds. It is also unwise to assume that any part of addressible memory space is 'more likely to be used by disk cache' or for any other purpose. You can only say that of parts of the old 640k DOS reserved area.

Program integrity checks are performed by your OS constantly. For instance CRCs tell Windows when important areas like /Drivers and /ServicePackFiles, etc., are altered. Individual subsystems like disk controllers run their own independent hardware integrity tests. There are already enough built-in checks and balances simply because any persistent failure in such a crucial area would put a vendor out of business. You can rest assured that you will soon know if you have a stuck bit.

The fact is literally countless billions of copy-without-extra-validation operations happily occur safely on hundreds of millions of computers around the globe every second without ever suffering problems of the kind claimed in this thread. That would seem to me to evidence enough that yet another layer of checks is utterly unnecessary except for the most extreme of cases which truly cannot tolerate any failure whatsoever. (You're not running a nuclear-tipped cruise missile fire-control system from your Dell laptop I hope?)