This forum uses cookies. Click X button to hide this message. What is stored? 
Total Commander Forum Index Total Commander
Forum - Public Discussion and Support
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Wrong SHA-512 on large files in some cases

 
Post new topic   Reply to topic    Total Commander Forum Index -> TC9.0x bug reports (English) Printable version
View previous topic :: View next topic  
Author Message
AndrewCreator
Member
Member


Joined: 10 Oct 2011
Posts: 121

PostPosted: Fri Apr 21, 2017 3:14 am    Post subject: Wrong SHA-512 on large files in some cases Reply with quote

Total Commander 9.0a on Windows Server 2003 32-bit

Run as administrator to instantly create test files with size 511 and 513 MiB:
Code:
fsutil file createnew test511.file 535822336
fsutil file createnew test513.file 537919488


Calculate SHA-1 and SHA-512 checksums:
Code:
ad4ea886fa65df3ae5ee0285bf8fde502d9014c5 *test511.file
56b0894b2b968f40e7a5e9240d242c19f7c56b70 *test513.file

443c60e846b9beb6b2e0f081b8cbe20959fc227a37d1539476ea6a6acf2a4f896a2ffba11a91438c03b8d6cb6d63f81002b3764c0e0f02583836615c714c9f9c *test511.file
52e70596311bad8f27d21734fedf5ad00c53fcb13d01b506d8285f29de85ab69be940506b4b08ca7ed7514e290b31c59cf6d2364f8d8236534aa82d28991d0f3 *test513.file <=== WRONG!


Repeat the same on Windows 8.1 64-bit:
Code:
ad4ea886fa65df3ae5ee0285bf8fde502d9014c5 *test511.file
56b0894b2b968f40e7a5e9240d242c19f7c56b70 *test513.file

443c60e846b9beb6b2e0f081b8cbe20959fc227a37d1539476ea6a6acf2a4f896a2ffba11a91438c03b8d6cb6d63f81002b3764c0e0f02583836615c714c9f9c *test511.file
4ebb0834b14026bb6e37c7b667817c640a4a1154fc0b146232155207eb938cba7ee73377be756c7773886685457115160386adf3d8fd395246c75fe5ba72250a *test513.file


SHA-512 for 513 MiB file is wrong on Windows Server 2003 32-bit while SHA-1 is correct.

SHA-384 is also wrong. SHA-256 is correct.
Back to top
View user's profile Send private message
Dalai
Power Member
Power Member


Joined: 28 Jan 2005
Posts: 5372
Location: Meiningen (Südthüringen)

PostPosted: Fri Apr 21, 2017 5:57 pm    Post subject: Reply with quote

I can not confirm this. I did the following in TC 9.0a on XP (32-bit) and Win7 (64-bit): Used your commands to create the files and told TC to calculate hashes: first SHA-1, then SHA-512, then SHA-384. My hashes match the ones you got on Win8.1.

I admit that XP is not Server 2003, but you should verify that you don't have a bad sector on your 2003 box, maybe create the files on a different disk or a USB drive (to rule out HDD controller drivers).

Regards
Dalai
_________________
#101164 Personal licence
Athlon II X4 640, 4096 MB RAM, Gigabyte M56S-S3, WinXP SP3

Plugins: Services2, Startups
Back to top
View user's profile Send private message Send e-mail
Gral
Senior Member
Senior Member


Joined: 26 Jan 2005
Posts: 466

PostPosted: Mon Apr 24, 2017 3:10 am    Post subject: Re: Wrong SHA-512 on large files in some cases Reply with quote

I can confirm bug on Windows Server 2003!
Exactly the same results - exactly the same (wrong) cheksum for 513 MB file.
Bug does not occurs on Windows XP!

2 AndrewCreator
Please post more info: windows edition, ram, CPU.

(EDIT)
This bug does not occurs with TC 8.52a!
Back to top
View user's profile Send private message
Dalai
Power Member
Power Member


Joined: 28 Jan 2005
Posts: 5372
Location: Meiningen (Südthüringen)

PostPosted: Mon Apr 24, 2017 6:15 am    Post subject: Reply with quote

I just tried this on a VM running Server 2003 SP1 32-bit (yes, that old Wink), and now I can confirm the bug. The bug is present at least since TC 9.0 beta4 which is the oldest I tested.

So, to summarize, the bug only occurs if all of the following conditions are met:
  • TC 9.x on Windows Server 2003
  • hashes SHA-512 or SHA-384
  • Specific file size, 513 MiB in this example
The verification is also affected, i.e. the checksum files with the wrong checksums are found to be correct by TC.

Regards
Dalai
_________________
#101164 Personal licence
Athlon II X4 640, 4096 MB RAM, Gigabyte M56S-S3, WinXP SP3

Plugins: Services2, Startups
Back to top
View user's profile Send private message Send e-mail
ghisler(Author)
Site Admin
Site Admin


Joined: 04 Feb 2003
Posts: 33621
Location: Switzerland

PostPosted: Mon Apr 24, 2017 7:16 am    Post subject: Reply with quote

TC9 uses Microsoft Crypto API to create these checksums, which is much faster than the internal functions used by TC 8.52a. It sounds like the crypto api functinos are faulty on Server 2003 - very odd.
_________________
Author of Total Commander
http://www.ghisler.com
Back to top
View user's profile Send private message Send e-mail Visit poster's website
chandragor
Junior Member
Junior Member


Joined: 01 Jun 2005
Posts: 58
Location: Italy

PostPosted: Wed Apr 26, 2017 8:22 am    Post subject: Reply with quote

Maybe this could be the reason:

[https://support.microsoft.com/it-it/help/938397/applications-that-use-the-cryptography-api-cannot-validate-an-x.509-certificate-in-windows-server-2003]
"This problem occurs because the Cryptography API 2 (CAPI2) in Windows Server 2003 does not support the SHA2 family of hashing algorithms. CAPI2 is the part of the Cryptography API that handles certificates."
_________________
Happy owner of license #12422 since 1997
Back to top
View user's profile Send private message
AndrewCreator
Member
Member


Joined: 10 Oct 2011
Posts: 121

PostPosted: Wed Apr 26, 2017 10:56 am    Post subject: Reply with quote

2chandragor, I have checked this. crypt32.dll on my system is newer, so there is another reason.

I have checked on Windows Server 2003 64-bit, the problem remains.
Back to top
View user's profile Send private message
Dalai
Power Member
Power Member


Joined: 28 Jan 2005
Posts: 5372
Location: Meiningen (Südthüringen)

PostPosted: Wed Apr 26, 2017 1:46 pm    Post subject: Reply with quote

Mmh, the issue is a little bit larger. After reading Andrew's note on Windows Server 2003 64 bit I immediately thought of Windows XP x64 which uses the same kernel. So I made the same tests on two XP x64 machines and can confirm the issue there as well.

So this problem occurs on Windows Server 2003 (both 32 and 64 bit) and Windows XP x64 - or, in short, Windows NT 5.2. It's really weird that NT 5.1 is not affected, and NT 6.1 is not affected. Don't know about NT 6.0, though (I don't have access to a Vista machine).

Regards
Dalai
_________________
#101164 Personal licence
Athlon II X4 640, 4096 MB RAM, Gigabyte M56S-S3, WinXP SP3

Plugins: Services2, Startups
Back to top
View user's profile Send private message Send e-mail
ghisler(Author)
Site Admin
Site Admin


Joined: 04 Feb 2003
Posts: 33621
Location: Switzerland

PostPosted: Thu Apr 27, 2017 3:29 am    Post subject: Reply with quote

I have checked it too now - it IS a bug in crypto API also in Windows XP!

It only occurs with files > 512 MByte. When creating a SHA2 hash, the file is split into blocks and the hash function is appied to each block. At the end, the last block is filled with zeroes followed by the number of bits (not bytes!) in the file. 8*512M = 4GBit. This causes the counter to go from 4 bytes to 5 bytes. Apparently this wasn't implemented correctly by Microsoft in Windows XP. TC 8.x uses an internal SHA2 library where it is implemented correctly.

Unfortunately XP is out of support, so the function will never be fixed by Microsoft. I will therefore have to switch to the slower internal SHA 2 functions in the next version of TC for files >512MB on XP/Server 2003...

Btw, I tried to find out what exactly Windows XP is doing wrong by setting various invalid values for the counter, but unfortunately without success. I will add a switch to wincmd.ini so users can still calculate the bad checksum for verifying checksums created with TC9.
_________________
Author of Total Commander
http://www.ghisler.com
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Dalai
Power Member
Power Member


Joined: 28 Jan 2005
Posts: 5372
Location: Meiningen (Südthüringen)

PostPosted: Thu Apr 27, 2017 4:34 am    Post subject: Reply with quote

ghisler(Author) wrote:
I have checked it too now - it IS a bug in crypto API also in Windows XP!

But why can I create correct SHA-512 checksums with TC 9.x on my XP systems even with files sized 4.7 GB (DVD ISO)? Doing the same on XP x64 or Server 2003 (x86) results in bad checksums.

Regards
Dalai
_________________
#101164 Personal licence
Athlon II X4 640, 4096 MB RAM, Gigabyte M56S-S3, WinXP SP3

Plugins: Services2, Startups
Back to top
View user's profile Send private message Send e-mail
ghisler(Author)
Site Admin
Site Admin


Joined: 04 Feb 2003
Posts: 33621
Location: Switzerland

PostPosted: Mon May 01, 2017 7:15 am    Post subject: Reply with quote

Apparently they don't use the same library...
_________________
Author of Total Commander
http://www.ghisler.com
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Dalai
Power Member
Power Member


Joined: 28 Jan 2005
Posts: 5372
Location: Meiningen (Südthüringen)

PostPosted: Thu May 04, 2017 7:40 am    Post subject: Reply with quote

Well, then I don't get why you said that the bug also exists in WinXP (32-bit). I mean, I don't think that calculating the hash the wrong way can result in the correct hash. Or am I missing something?

Regards
Dalai
_________________
#101164 Personal licence
Athlon II X4 640, 4096 MB RAM, Gigabyte M56S-S3, WinXP SP3

Plugins: Services2, Startups
Back to top
View user's profile Send private message Send e-mail
ghisler(Author)
Site Admin
Site Admin


Joined: 04 Feb 2003
Posts: 33621
Location: Switzerland

PostPosted: Mon May 08, 2017 7:44 am    Post subject: Reply with quote

It does exist in TC 32-bit on XP 64-bit here...
_________________
Author of Total Commander
http://www.ghisler.com
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Dalai
Power Member
Power Member


Joined: 28 Jan 2005
Posts: 5372
Location: Meiningen (Südthüringen)

PostPosted: Mon May 08, 2017 9:14 am    Post subject: Reply with quote

Well, I did say the exact same thing, didn't I? When someone says something about WinXP I assume that he/she means WinXP 32 bit, not XP x64 (the latter didn't have a relevant market share). I guess you didn't mean XP 32 bit but XP x64 - which corresponds to my findings.

So, to summarize (again), and to avoid further confusion: NT 5.1 is NOT affected, only NT 5.2 - Windows Server 2003 x86/x64 and XP x64 - is.

Regards
Dalai
_________________
#101164 Personal licence
Athlon II X4 640, 4096 MB RAM, Gigabyte M56S-S3, WinXP SP3

Plugins: Services2, Startups
Back to top
View user's profile Send private message Send e-mail
Display posts from previous:   
Post new topic   Reply to topic    Total Commander Forum Index -> TC9.0x bug reports (English) All times are GMT - 6 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Impressum: This site is maintained by Ghisler Software GmbH

Using phpBB © 2001-2005 phpBB Group