Verify after copy

English support forum

Moderators: Stefan2, white, sheep, Hacker

Post Reply
Sob
Power Member
Power Member
Posts: 908
Joined: 2005-01-19, 17:33 UTC

Post by *Sob » 2010-07-12, 23:00 UTC

SQUIRE wrote:Always on/off bit on an otherwise correctly functioning system? Your POST will usually immediately catch it on boot up. ...
I had few bad RAM sticks for free from friendly computer store back in the days when 128MB was a lot of memory and quite expensive. Just few bad bytes at static addresses. They worked very well for experiments in Linux with BadRAM patch. Did POST ever complain? Not even once.
Did all this integrated error checking help me last month when my harddrive decided to entertain me with some unreadable sectors? S.M.A.R.T.'s raised Current Pending Sector attribute was really helpful after the data failed to read. Would re-reading the file just after write help? Possibly yes.
Or did I get any indication about something going wrong few years ago, when after applying SP1 to Win2k3, something started to silently corrupt SMB transfers from clients to server? Would it allow for saving backups for unlimited time before it was discovered? Why not, as long as no one needed to restore some data.
Program integrity checks are performed by your OS constantly. For instance CRCs tell Windows when important areas like /Drivers and /ServicePackFiles, etc., are altered.
Well, yes, Windows will detect some changes, to e.g. signed drivers, but it's only very small portion of files. Try to open your notepad.exe in hexeditor and change few random bytes, I mean really random in the middle of the file, not right the "MZ" at the beginning. Not only will the changed executable most likely run just fine, but there won't be any warning from OS about changed file. So much for corruption not being able to stay hidden for too long.
The fact is literally countless billions of copy-without-extra-validation operations happily occur safely on hundreds of millions of computers ... That would seem to me to evidence enough that yet another layer of checks is utterly unnecessary...
I'm not sure, maybe once you're so unlucky that you lose some data because of this kind of rare error, it makes you paranoid for the rest of your life. ;) Some people just feel that some kind of verification is needed. No one would force you to use it. TC doesn't even have to completely support it by itself, some simple support for hooking custom verification to file operations could be enough.
(You're not running a nuclear-tipped cruise missile fire-control system from your Dell laptop I hope?)
I'd of course like to, but try to find nuclear-tipped cruise missile somewhere cheap. And with all those anti-nuclear stances in the world, they would surely try to put embargos on my country, you know how it goes. It's just not worth it. ;)

User avatar
SQUIRE
Senior Member
Senior Member
Posts: 373
Joined: 2005-06-16, 18:07 UTC

Post by *SQUIRE » 2010-07-13, 02:09 UTC

@ Sob:
Did POST ever complain? Not even once...
Well, I'm sorry you've had such bad experiences but I have to point out that your store did indicate that they knew the RAM was faulty. How would they have known if not alerted by some variant of a POST-style memtest in the first place? Second, if your POST failed to detect a proven bad stick then you urgently need to ask your BIOS maker for an explanation. Since the store's system found the fault and your system didn't, does it not suggest something amiss somewhere? You can't both be right.
my harddrive decided to entertain me with some unreadable sectors...
Er, I would say the error checking did its job perfectly well when it told you the sectors were unreadable. If it had lied and said everything was just fine then it could be said to have not worked. What were you expecting it to say? Surely it is asking rather a bit much to have the controller look into the future to accurately predict, "Hey, watch out user, these sectors are OK at this moment but are gonna fail at 10 am tomorrow when you try to read them", isn't it? Do you really think that a copy validation program could have saved you?
Well, yes, Windows will detect some changes, to e.g. signed drivers, but it's only very small portion of files...So much for corruption not being able to stay hidden for too long.
Is Notepad a critical system component? Nope, just another humble editor. Obviously MS quite rightly considered it unnecessary to protect with the occasional quick hash once-over. On the other hand Ghisler doesn't approve of TC being tampered with and if you do what you suggested you will be told that the program is corrupt. So it isn't a question of checks not working, it's whether and how the maker chooses to deploy them. That's just as well as given the vast number of different files that make up an OS I would hate to think what impact it would have if Windows insisted on periodically hash checking every single one just in case.
...it makes you paranoid for the rest of your life.
Well, I suppose if your trousers did fall down one time in public in front of everyone you'd be so traumatised that you might insist on wearing both a belt AND braces for protection forever more, when rationally just the usual zip would have worked fine as it does for most everybody. It's entirely up to Ghisler if he wants to supply you with the belt and braces or not, of course. I wouldn't hold my breath while waiting if I was you.

Buy the missile from N. Korea and tell Bill Gates, "Add an automatic copy-verify routine into Windows or Redmond gets it!" That ought to get MS' attention one way or another. Don't bother threatening the Penguin guys. They'll just go, "Hey, looky here guys, a real nuke! Cool! I wonder what its blast radius is?".

Sob
Power Member
Power Member
Posts: 908
Joined: 2005-01-19, 17:33 UTC

Post by *Sob » 2010-07-13, 03:59 UTC

SQUIRE wrote:Well, I'm sorry you've had such bad experiences ...
It was in fact good experience for me in that RAM case, as I had plenty of memory for playing and for free. :) Sure it was faulty, but it was possible to work around it reliably enough for experimental usage. I don't know how the errors were originally detected. Maybe POST did it. Or maybe not, often such problems are found by specialized software like Memtest86. And the fact that someone bothers to create such software suggests that POST may not be so good at catching memory errors, doesn't it?
Surely it is asking rather a bit much to have the controller look into the future to accurately predict, "Hey, watch out user, these sectors are OK at this moment but are gonna fail at 10 am tomorrow when you try to read them", isn't it?
I'm sold, give me ten of those right away! :)
Do you really think that a copy validation program could have saved you?
It depends entirely on when those sectors became unreadable. If it was immediately after they were written, then yes, validation would have saved me. It would try to read the file, fail and say "Oops, something went wrong, don't even think about deleting the source file yet". Well, not in this case, because I didn't move the file but rather unpacked it from archive on another disk and didn't delete the archive. I actually got to using that file the next day and found the problem. So I was quite lucky.
Is Notepad a critical system component?
Notepad was just randomly chosen ordinary application, to show that if few bytes in file get changed for any reason, it doesn't necessarily have to manifest itself.
I wouldn't hold my breath while waiting if I was you.
I'm not. I can live with it. I don't have to verify everything, it's the advantage of being only semi-paranoid. ;) When I feel the need to verify something, it's not that hard to do it manually. But if TC helped me with it in any way, I wouldn't say no.

User avatar
hlloyge
Member
Member
Posts: 120
Joined: 2006-11-02, 23:14 UTC

Post by *hlloyge » 2010-10-10, 06:47 UTC

Well, to add to a topic - I've been doing some large copying to a Dell server, 2 gigs of ECC RAM, CentOS 5.5 installed, through FTP. When I deploy new file server, I usually do some checking after copying by the means of md5 files, and I found out something interesting - every file above 1 GB was corrupt. Instead of bashing TotalCMD for the non-existent CRC-after-copy function, I went for a hunt for a reason for this - checking for bugs in new vsftpd did nothing. Booting the server and watching POST revealed nothing. I asked a coleague did he do something with the server before giving it to me, and he admitted adding one gig of memory :) we removed it (and later tested it, sure, it was faulty), and the problem was gone. Tested remaining RAM showed no problem.
Fortunately, there were no business-critical data there :) but what is curious is that my coleague was running Windows server 2008 for months without any problems, and CentOS was also running without any suspicious behaviour.
To conclude to what I said before, when having problems with data integrity, hardware is faulty. That is why you have to have BACKUPS, and regularly check hardware health.

User avatar
Sheepdog
Power Member
Power Member
Posts: 5150
Joined: 2003-12-18, 21:44 UTC
Location: Berlin, Germany
Contact:

Post by *Sheepdog » 2010-10-10, 22:35 UTC

SQUIRE wrote: Always on/off bit on an otherwise correctly functioning system? Your POST will usually immediately catch it on boot up.
Nowadays in most PC BIOSes the Quick Power On Self Test is enabled what means that the memory test will be skipped.
And if it is performed, because some user knows about it and finds it worthy and disabled the 'Quick' - only very few failures are found.
To find all possible failure on memory you have to write and read many different patterns to each memory cell - what takes much longer as the usual PC user is willing to wait (till his Windows is running).

That for are tools like memtest86 that perform a complete memory test - and if wished for several times. You should possibly let memtest86 check 2Gb RAM to see how long this would take and compare this time with the POST.

Apart from that I'm convinced that copy errors usually are caused by defective hardware. And if I would be faced with as many copy errors as described in this thread I really would try to improve my hardware.

But if that would not be possible, I think a verify function would be highly desirable.

Personally I use CRC-check always after burning a CD/DVD using the DVDSIG.

And I'm not against a 'Verify' checkbox (or ini-option)

Sheepdog
"A common mistake that people make when trying to design something
completely foolproof is to underestimate the ingenuity of complete fools."
Douglas Adams

User avatar
luckylittle
Junior Member
Junior Member
Posts: 8
Joined: 2010-10-19, 13:14 UTC

Post by *luckylittle » 2010-10-20, 13:37 UTC

I am using 3rd party software called TeraCopy that does the trick.

smartwork
Junior Member
Junior Member
Posts: 5
Joined: 2010-11-07, 09:38 UTC

Post by *smartwork » 2010-11-07, 09:50 UTC

8 Threadpages of discussion for a pretty simple feature. I am using Total Commander (or formerly Windows Commander) now for about 10 years, its always one of the first things I install on a new system and a great tool without any question. So it's absolutely not understandable for me that seeing how it has improved and got new features added over the years there has to be a BIG discussion about a such a SIMPLE feature request.

When packing ZIPs they are verified too after packing - so why here but not on copies? I've had all types of data loss over the years, memory errors out of the blue because the slot (!) simply stopped working propperly, (rarely) unstable memory, network errors, bad sectors and such. I often thought "if there would only be a verify...". Now I am really thinking to look for another software to perform large copies because it really sucks to have to sync/compare manually everytime.

So I want after those 8 page of discussion ask, are there still no plans to add a verify checkbox (or even default setting) for copy/move operations?

emerson
New Member
New Member
Posts: 1
Joined: 2010-12-28, 15:46 UTC

Verify after copy

Post by *emerson » 2010-12-28, 16:05 UTC

Hi
Just wanted to check if TC do support byte by byte comparision when moving files yet?

I'm not using TC, but will consider buying TC if the copy/move with verify was supported. I saw this thread long while ago and I was hoping the feature was supported by now.

I'm currently using Windows explorer to copy files (200 - 500 GByte) over the network and then use NoClone to verify that the files are correct before removing the original files.
Time is not an issue, since this is done when we are not using any of the computers.

Do anyone know of a tool that can handle copy/verify/remove in a better way?


Thanks
/E

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 37570
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) » 2010-12-30, 14:09 UTC

No, currently not, sorry. "Synchronize dirs" allows to compare by content after copying, but this doesn't guarantee another read from disk - it may come from the write cache. The idea of the compare step in "Synchronize dirs" isn't meant to make an integrity check, but to check which files are different to select them for copying
Author of Total Commander
http://www.ghisler.com

User avatar
Phred
Senior Member
Senior Member
Posts: 272
Joined: 2009-06-16, 15:24 UTC
Location: SEAu

Read While Writing

Post by *Phred » 2013-06-21, 19:08 UTC

I've noticed that the VLC player can start playing audio podcast files that are still being downloaded from a web source.
And we all know that a DVR, a digital video recorder, can, among other things, play an earlier part of a TV program whose current broadcast stream is being recorded - recorded to the latter end of the same file from which content is being played back.

I'm musing that perhaps a child TC process could begin reading the early sectors of a target file its parent has just laid down - while the parent process continues writing the file's remainder - and compare them with its companion source.

Like a second sweeper sweeping immediately behind a leading sweeper; the combination of the two would likely finish in much the same time as only one.

The copying/moving process would have as its greatest bottleneck the target object being written to, so a target read, pathway transfer (being either cache, disk sub-system or a bi-directional network), plus a comparison, would likely be possible to execute simultaneously. CPU power is likely to be available in quantity.

The source would probably be still lying in cache, but whether it is re-read from cache or read from the source location, it shouldn't matter: reads are quicker than writes.

Summary: while a file is being written, a second process follows close behind comparing target-so-far to the source, before the writing process is finished.

I don't know, but a background simultaneous verification function would seem to me to be eminently possible, without the operation taking inordinately longer than without verification at all.
Yes, I understand that deeper sub-systems act to make this function virtually unnecessary, but having this extra layer at the user level would compliment the TC product extremely well, IMHO.
Discuss, if you will.

hakapes
Junior Member
Junior Member
Posts: 5
Joined: 2011-11-27, 20:33 UTC

Post by *hakapes » 2013-07-25, 11:50 UTC

Is there any improvement on verify after copy?

Unfortunately I have lost about 50GB of backup data, I moved from one hard drive to a new one, on the directory level it seemed to be good.

However, later on, I discovered that the files were actually empty, somehow they didn't copy.

User avatar
hlloyge
Member
Member
Posts: 120
Joined: 2006-11-02, 23:14 UTC

Post by *hlloyge » 2013-08-05, 13:30 UTC

hakapes wrote:Is there any improvement on verify after copy?
Unfortunately I have lost about 50GB of backup data, I moved from one hard drive to a new one, on the directory level it seemed to be good.
However, later on, I discovered that the files were actually empty, somehow they didn't copy.
Somehow, you have more serious problems than missing option from your favorite file manager.

Thany
Senior Member
Senior Member
Posts: 251
Joined: 2003-09-30, 09:20 UTC
Location: Netherlands

Post by *Thany » 2014-03-08, 15:51 UTC

Wow, that's a topic from long ago. I stumbled upon this topic just now because I too was looking for a "verify after copy" tick. I was amazed it's not there. Am I missing something?

Seeing the things TC can do, verify after copy should be fairly trivial.

Here's how it works:

1. Copy a file
2. Read the copied file (from disk, not from the filecache!)
3. Compare byte-by-byte against source

It wouldn't work with move, no. Of course it wouldn't.

User avatar
HolgerK
Power Member
Power Member
Posts: 5080
Joined: 2006-01-26, 22:15 UTC
Location: Europe, Aachen

Post by *HolgerK » 2014-03-08, 19:53 UTC

Did you miss TC 8.50?

Copy dialog -> [_] Verify (directly above the "Options" button)

BTW: still unchecked here.

Regards
Holger

sky66
Member
Member
Posts: 107
Joined: 2014-02-22, 08:44 UTC

Post by *sky66 » 2014-03-08, 23:38 UTC

Configuration => Options... => Copy/Delete => [_] Verify after copying (MD5 checksum)

Refer to TOTALCMD.CHM:
Reads the copied file again after copying finishes, and compares its MD5 checksum with the original. The disk cache will be bypassed. Since this function is slow, it should only be used if there are any known problems with disks. This checkbox determines the state of the option when Total Commander starts. It can also be changed directly in the F5 Copy dialog.

Post Reply