Generate MD5 log for all copied files

[isidro]

It would be VERY useful to have MD5 computed while copying and have results on one MD5 file on destination (name should be source upper folder/drive). That would provide an integrity verification on destination media to ensure backup was done ok. Current Verify after Copy isn't useful because most files will probably be read from Cache instead and future integrity check can't be done.
Now my backups consists of four steps: 1) copy all. 2) Generate MD5 in source. 3) Generate MD5 on dest. 4) compare generated MD5 files to make sure nothing got corrupted.
With MD5 checkbox on copy it will be reduced to 2 steps: Copy+MD5 and use that MD5 file to verify dest. Faster an simpler and having MD5 will ensure integrity in the future.

[Dalai]

This would indeed be useful. Support++ for the request itself.

One point of discussion is: Where exactly integrate this feature in the copy dialog? Where to put the type of checksum choice? Sometimes I don't want MD5 but SHA1 or SHA256 instead. And what about the different style of checksums, all checksums in one file vs. one checksum file per file?

@Mods: This thread should be moved to TC Suggestions subforum.

Regards
Dalai

[isidro]

Use last type of checksum used by user. Checksums should be per root folders or per full set and not per file (ie: bakups of 10k mp3, flacs or photos will stupily duplicate number of files). It should work exactly as Pack works now with: "Create separate archives, one per selected file/dir"

[MVV]

You have to add a step into your algorithm: 2) flush target device cache. Otherwise your OS may simply get data from cache on your step 3).

[l0o]

Hello!
Any update to this idea?
I'd like to have such option (generate md5 is just by the way - during moving, or copying) for future check. It would be very useful!

Where exactly integrate this feature in the copy dialog?
Maybe where "Verify after copy" is? "Save md5 of copied/moved files" or smthng like that.

Where to put the type of checksum choice? Sometimes I don't want MD5 but SHA1 or SHA256 instead.

Well anyway better to have MD5 than nothing and those precious checksums wouldn't vanish after compute.

And what about the different style of checksums, all checksums in one file vs. one checksum file per file?

Let's keep it simple: All in one file saved to destination folder. Assume it would be useful for moving big data (to have verify possibility in future).

[isidro]

You have to add a step into your algorithm: 2) flush target device cache. Otherwise your OS may simply get data from cache on your step 3).

That would be true on a small copy, in my case usually I copy between 1Tb and 6Tb, there's no way for cache holding that amount of data, so I don't bother flushing. I had detected bad memory chips due to one MD5 differences every 300Gb or so. My data would have been gradually corrupted if MD5 checks weren't done.
Anyway, MD5 of source copied files on destination would save me 3 hours of reading again the same 6Tb.

[Goblin_NN]

be sure to md5? try https://github.com/BLAKE3-team/BLAKE3

Spoiler

Image: https://github.com/BLAKE3-team/BLAKE3/raw/master/media/speed.svg

[isidro]

be sure to md5? try blake3

Irrelevant to this topic. And my bottleneck is drive speed and not checksum compute time.

[isidro]

be sure to md5? try https://github.com/BLAKE3-team/BLAKE3

Spoiler

Image: https://github.com/BLAKE3-team/BLAKE3/raw/master/media/speed.svg

Blake is better, but still same problem: unnecessary reading the whole HD twice: one for Hash gereration, and another when copying.
[Having the verification results written to a text file would be very useful as well.] >>> Just discovered as a good workaround that all items can be marked and then copy pasted.

[nemoW]

I strongly support this feature. Didn’t find any Windows utilities that could create a file checksum while copying

[white]

Moderator message from: white » 2024-03-13, 10:54 UTC

Fixed the links in these posts:

[JOUBE]

I strongly support this feature. Didn’t find any Windows utilities that could create a file checksum while copying

This is not a good idea.

Checksuming the target via the source is always a "self-fulfilling prophecy".