Store CRC hashes (e.g. MD5) in ADS (Alternate Data Streams)

[knn]

Total Commander has the great function to calculate file hashes, e.g. create/verify the checksum with the method MD5, CRC32, SHA256 etc.

Is it possible to store those checksums in the ADS (Alternate Data Stream)?

[ghisler(Author)]

No, this isn't supported, sorry. Why? There is no standard, e.g. for the name of the ADS and the contents, so it would only work within Total Commander. The checksum files I use are all using a standard format (SFV for CRC, regular checksum files as used on Linux for all others).

[knn]

No, this isn't supported, sorry. Why? There is no standard, e.g. for the name of the ADS and the contents

I don't think that's a big disadvantage. Simply invent a standard, you will become more famous than you already are, the Ghisler Hash Standard will get its own Wiki page . I suggest to use "hash.md5", "hash.sha256" as stream name to store the hash. One could also use "hash.json" to store it in JSON format, e.g.

Code: Select all

{"timelastmodified":1657802239,"hashes":{"md5":"6cd3556deb0da54bca060b4c39479839","crc32":3957769958}}

so it would only work within Total Commander.

I don't think that would be a problem.

[Dalai]

Theoretically it would be possible. But the point of having checksums is to be able to verify them, and also to easily edit them. How do you verify something you can't see without a third-party application? Even in TC you need a plugin to be able to tell if a file has an ADS, and how it's called.

When copying files to file-system which don't support ADS, e.g. FAT32 on a USB flash drive, or UDF on optical media, the checksums won't be available. Even worse, when moving files to such file-system, the checksums are gone entirely - without warning.

In short: I don't see the benefit, but a couple of serious drawbacks.

Regards
Dalai

[knn]

Theoretically it would be possible. But the point of having checksums is to be able to verify them

Those functions already exist in TC. My suggestion is merely to be able to store checksums in the ADS. TC is already able to write a file, I merely suggested to store them in the ADS, so that copying, renaming etc keeps the checksum available.

When copying files to file-system which don't support ADS, e.g. FAT32 on a USB flash drive, or UDF on optical media, the checksums won't be available. Even worse, when moving files to such file-system, the checksums are gone entirely - without warning.

That's a general issue with ADS. In such cases TC could have an option to copy ADS's content to an *.md5 file if needed. No problem. Moreover it's a marginal issue since TC already has a verify function (to check whether a copy is correct). Having is a checksum stored in the ADS is used to be able to verify it after years, hardly anyone uses USB sticks as archives that have to last for decades.

[Dalai]

TC is already able to write a file, I merely suggested to store them in the ADS, so that copying, renaming etc keeps the checksum available.

The checksums are kept available when copying or renaming files, though renaming is a special case because the checksum file needs to be edited to reflect the new name. And there's the next issue because you can't just hit F4 like you would do when having checksum files. I still don't see any advantage in storing the checksums this way. Yes, copying might be easier because there are less files to select. But is that really a problem? How often do you forget to copy checksum files?

In such cases TC could have an option to copy ADS's content to an *.md5 file if needed. No problem.

Then what's the point of having the checksums in ADS when they end up in files again (in some cases) anyway? And yes, it is a point that needs to be discussed beforehand.

There are even more things that need to be considered:

• What about packing files to archives? Many archive formats don't support ADS. RAR does, but that's the only one that comes to mind. WIM is another one, but that's not really an archive format like ZIP and RAR.
• What about transferring files to other systems that don't support ADS? How to verify checksums there? Checksum files are more or less platform independent, ADS are not by a long shot.

I'm sorry if my answers come across as negative. Often times it's not as simple as it seems to be at first glance.

Regards
Dalai

[knn]

Those are general issues with ADS and have not much to do with what I want. Your listed issues apply to other things as well, such as file size limits, file naming conventions, file encryption and extended attributes. Those issues are merely limitations of file systems and have not really anything to do with my suggestion.

Also, one does not need to edit checksums. TC does not need to offer any ability to manually edit checksums.

[Dalai]

Those are general issues with ADS and have not much to do with what I want.

How so? You suggested to save checksums in ADS, and the issues listed above apply to ADS. How come that it doesn't have much to do with it? If you'd say, "those issues are not relevant to me", that's fine. But the points I listed are relevant nevertheless, maybe not for you but most likely for other people.

Also, one does not need to edit checksums.

Says who? I've edited checksum files quite often in the past couple of years. It's more about the file names or whole lines and less about the checksums themselves. Renamed some files that are already in checksum files? Need to remove some files/lines from the checksum files because the've been deleted? Need to convert a checksum file's line endings? In all of these cases it's necessary to edit such files. And the list goes on. And I've replaced the checksum of a bad file, but only once or twice.

If you didn't need any of these things so far, then good for you. Other people have different needs.

TC does not need to offer any ability to manually edit checksums.

Indirectly it does because you can just edit the checksum files in the editor of your choice. ADS can be opened in an editor, too. The way to go there is much longer because you can't see the ADS, so you'll need to know its name and still can't just press one key to open it in an editor. Well, it's probably possible, but it's so much more hassle.

I still don't see the little advantage of copying the checksums together with the files (but not in all cases!) outweighing the drawbacks I listed above. But maybe I don't have a wide enough view, who knows.

Regards
Dalai

[Hacker]

Dalai,

Those are general issues with ADS and have not much to do with what I want.

How so? You suggested to save checksums in ADS, and the issues listed above apply to ADS. How come that it doesn't have much to do with it? If you'd say, "those issues are not relevant to me", that's fine. But the points I listed are relevant nevertheless, maybe not for you but most likely for other people.

I think what the OP meant is that we already have the same NTFS / ADS issues and nobody says not to support NTFS / ADS features just because they might get lost when moving to another file system.

Also, one does not need to edit checksums.

Says who? I've edited checksum files quite often in the past couple of years. It's more about the file names or whole lines and less about the checksums themselves. Renamed some files that are already in checksum files? Need to remove some files/lines from the checksum files because the've been deleted? Need to convert a checksum file's line endings? In all of these cases it's necessary to edit such files. And the list goes on. And I've replaced the checksum of a bad file, but only once or twice.

Well, if the checksums were in ADSs, we wouldn't need to edit them in the sense you are describing (e.g. rename).

Not saying I support this wish but let's make our argument correct.

Roman

[Dalai]

I think what the OP meant is that we already have the same NTFS / ADS issues and nobody says not to support NTFS / ADS features just because they might get lost when moving to another file system.

Well, it's not clear if the checksums in ADS are supposed to replace the checksum files, or if it's supposed to be an option, e.g. when creating checksums. The former wouldn't make any sense because of the issues I described so far. The latter makes me wonder how many people would use checksums that way.

Well, if the checksums were in ADSs, we wouldn't need to edit them in the sense you are describing (e.g. rename).

Why not? Let's say I have one checksum file for a directory. Now I move these files into a subdirectory. I need to edit the checksum file if I want to avoid having to recreate the checksums - which is what I want for large files. Should TC (silently) edit the paths in the ADS for me? What happens when a file gets overwritten, by TC or some other program? There are so many unknowns.

Regards
Dalai

[Hacker]

Dalai,

The latter makes me wonder how many people would use checksums that way.

I guess the checksums would be created and verified per file.

Let's say I have one checksum file for a directory.

My assumption was to have one checksum per file. My assumption might be wrong, though I guess it's the one that makes most sense regarding usability.

Roman

[Horst.Epp]

Dalai,

The latter makes me wonder how many people would use checksums that way.

I guess the checksums would be created and verified per file.

Let's say I have one checksum file for a directory.

My assumption was to have one checksum per file. My assumption might be wrong, though I guess it's the one that makes most sense regarding usability.

Roman

Checksums can only be for a file, NTFS ADS streams are for files only.

[white]

NTFS ADS streams are for files only.

Since when?

[Horst.Epp]

Sorry, wrong info
Even my tests with the NTFS_diz plugin and dirs are working.

[hi5]

Not very practical: but perhaps an alternative solution is to treat crc files the same was as descript.ion, so if set by the user, crcs are copied / moved with files - TC can already do this for file comments (decript.ion) and also saved web pages where the page + folder with css/images are moved/deleted together.

I don't think I would use it myself though.