Compare by content crashes for binary files >= 2GB

Bug reports will be moved here when the described bug has been fixed

Moderators: Stefan2, white, sheep, Hacker

Post Reply
User avatar
elgonzo
Senior Member
Senior Member
Posts: 377
Joined: 2013-09-04, 14:07 UTC

Compare by content crashes for binary files >= 2GB

Post by *elgonzo » 2018-01-07, 20:28 UTC

TC version: TC64 9.12, fresh INI file
OS: Windows 7 Pro x64, english

Comparing binary files with a size of 2 GB or larger was working fine with TC64 8.52a.

However, TC64 9.12 will crash when attempting to compare such files. (To be accurate, the crash happens for file sizes of 2GB-8 = 2147483640 Bytes and larger.)

Observing the drive active indicator LED and the growing program memory (aka working set) in the task manager, it looks like TC is actually loading and scanning the files.

After finishing the file scan (that is, TC's memory footprint not growing any further), TC fails to jump to the first difference. Instead, the "Compare by content" feature seems to enter some unproductive busy loop, causing a sustained CPU load of around 13% on my machine. TC itself remains usable, however the "Compare contents" window cannot be closed, and closing the TC main window will not exit the TC process due to the hanging "Compare contents" window.

Attempting to compare such files and letting TC idle with the hanging "Compare contents" window without any other user interaction will eventually lead to a crash of TC after a few minutes:

Code: Select all

Problem signature:
  Problem Event Name:	APPCRASH
  Application Name:	TOTALCMD64.EXE
  Application Version:	9.1.2.0
  Application Timestamp:	00000000
  Fault Module Name:	TOTALCMD64.EXE
  Fault Module Version:	9.1.2.0
  Fault Module Timestamp:	00000000
  Exception Code:	80000001
  Exception Offset:	0000000000029e81
  OS Version:	6.1.7601.2.1.0.256.48
  Locale ID:	2057
  Additional Information 1:	de35
  Additional Information 2:	de35b0e1f9a0fa23f716eeda67a9ae18
  Additional Information 3:	7ca2
  Additional Information 4:	7ca28b70ed2f7ef3c253510ae17aebe0
There is another observable issue when the file size is exactly 2GB or larger: The percentage in the status bar of the "Compare contents" window is not increasing (i.e., it remains at 0%), even though it appears that TC is loading/scanning the files.

The particular content of the files does not seem to matter (i tested with a number files i filled with random byte values, and where each pair of files had exactly two different bytes in two subsequent file positions). TC64 8.52a had no problem at all comparing and finding the differing bytes in these test files. For my tests, i used a fresh INI file for both TC64 9.12 and TC64 8.52a.

With the characteristic file size in mind (2GB-1 = 2147483647 = 0x0x7FFFFFFF is the max value of a signed 32bit integer), it looks to me like some of the new code in the "Compare by content" feature incorrectly uses 32bit data types/pointers/arithmetic where it should use 64bit data types...
Windows 7 Pro x64 SP1

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 36554
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) » 2018-01-08, 15:50 UTC

Unfortunately I cannot reproduce it - just tried with a 7.8 GB ZIP file. Are you trying to compare text files?
Author of Total Commander
http://www.ghisler.com

User avatar
elgonzo
Senior Member
Senior Member
Posts: 377
Joined: 2013-09-04, 14:07 UTC

Post by *elgonzo » 2018-01-08, 18:29 UTC

ghisler(Author) wrote:Unfortunately I cannot reproduce it - just tried with a 7.8 GB ZIP file. Are you trying to compare text files?
No. Binary files (there is nothing in those files that should make TC falsely assume text files). The "Compare contents" window also correctly uses the hex view mode.

Could it be that perhaps the issue might be related to where the first difference in the compared files occurs? In all my tests, the first difference was close to the end of the files. I will check what happens if i place the first file difference close to the beginning of the file and what happens when i use files as large as yours. I will then post my results here again.

By the way, did the percentage number / progress bar in the status bar increase when you tested with that 7.8GB file?
Windows 7 Pro x64 SP1

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 36554
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) » 2018-01-11, 10:30 UTC

Yes it did - but the differences were near the start. I will try with differences near the end.
Author of Total Commander
http://www.ghisler.com

User avatar
elgonzo
Senior Member
Senior Member
Posts: 377
Joined: 2013-09-04, 14:07 UTC

Post by *elgonzo » 2018-01-11, 18:33 UTC

I tested with a variety of very large files (around 7...8 GB) with random binary content, as well with very large 7zip and Zip archives of similar size. I did various experiments with placing differences very close to the start of the file (within the first 64KB), somewhere in the middle and sprinkling several differing bytes across the file. Unfortunately, no change in the observed behavior. :(

Using such large files, i noticed something that might further help pinpointing the bug: When attempting to compare very large files, TC64 9.12 memory allocation always stopped at a working set size of around 4400...4500 MB (~4.4 GB; and that's probably the point when TC goes off the rails, and which is also congruent with my suspicion i expressed at the end of my initial report).

In comparison, TC64 8.52 memory footprint grows according to the file size (with comparing 7.4 GB files, TC64 8.52 memory consumption would be a little more than 15 GB).

Also perhaps worth mentioning: As long as the memory used by TC64 9.12 is growing (i.e., has not reached ~4.4 GB), i am able to press the Abort button and thus prevent TC from going off the rails. If TC already found a difference at the time of me pressing the Abort button, the "Compare contents" window will also jump to the first difference.

Since TC64 8.52 works just fine on exactly the same system with exactly the same files that make TC64 9.12 hang/crash, i do believe the issue is inside TC64 9.12. (Heck, just to eliminate the remote chance of my TC installation being somehow corrupted, i even downloaded the 64bit version of 9.12 again, which i extracted to a new folder and used that for the tests.)

Weird that you can't reproduce it. What OS did you use? Win 7 x64 Pro like me, or something else? (Yes, i am grasping at straws now ;) )
Windows 7 Pro x64 SP1

umbra
Power Member
Power Member
Posts: 783
Joined: 2012-01-14, 20:41 UTC

Post by *umbra » 2018-01-12, 19:51 UTC

2elgonzo
If you can reproduce it in 9.12 and not in 8.52, maybe you could try to find out in which build it first appeared. There have been dozens of builds between those two versions and finding a specific build, or at least a smaller range of builds, could help ghisler to focus on specific code changes.
Windows 7 Pro x64, Windows 8.1 Pro x64, Windows 10 Pro x64

User avatar
elgonzo
Senior Member
Senior Member
Posts: 377
Joined: 2013-09-04, 14:07 UTC

Post by *elgonzo » 2018-01-12, 21:36 UTC

umbra wrote:2elgonzo
If you can reproduce it in 9.12 and not in 8.52, maybe you could try to find out in which build it first appeared. There have been dozens of builds between those two versions and finding a specific build, or at least a smaller range of builds, could help ghisler to focus on specific code changes.
Yeah, that's probably something i will have to do if Ghisler doesn't manage to reproduce the problem. Well, at least i could do it with regard to 9.00, 9.00a and 9.10 (as those are the only releases i can get from http://totalcmd.pl/pobierz/starsze).

I am not asking you to do something, but did you (or anyone else reading this) by any chance perhaps try reproducing the problem? It could help understanding better whether there are certain "environmental" conditions required to trigger the problem (like, for example, a specific OS version...)
Windows 7 Pro x64 SP1

browny
Member
Member
Posts: 166
Joined: 2007-09-10, 13:19 UTC

Post by *browny » 2018-01-13, 14:06 UTC

elgonzo wrote:did you (or anyone else reading this) by any chance perhaps try reproducing the problem?
I have sporadic crashes of the compare tool of 64-bit TC with small files, but never could reproduce that.
It was said that there is a "problem" with theme configuration, but that happens with unmodified default Windows theme in different OS versions.
I also inquired if the compare tool execution thread is protected with try...except block, but got no answer.

User avatar
petermad
Power Member
Power Member
Posts: 8073
Joined: 2003-02-05, 20:24 UTC
Location: Valsted, Denmark
Contact:

Post by *petermad » 2018-01-14, 01:30 UTC

2elgonzo
did you (or anyone else reading this) by any chance perhaps try reproducing the problem?
I have now tried with two 3 Gb zip files (one is a copy of the other where I just added one file).

In TC 8.52a, after the files are loaded and the CbC window is opened, TC starts counting differencies - during that period the % in the lower left corner increases and the progress bar in the lower right corner grows accordingly - after about 2 minutes it reaches 100% (115 differencies in the end of the files) and the "Abort" button in the upper left corner changes to "Compare". During the counting, Windows' cursor is in wait mode. If I during the differencies count press the "Abort" button, the counting stops and the button changes to "Compare".

In TC 9.12 I don't get a crash, but TC does NOT increase the percentage count and the progress bar does NOT show. Windows' cursor is in wait mode - and apparantly stays this way indefinitely (I observed it in many minutes). If I press the "Abort" button, the counting does NOT stop the process - I have to press Esc to make that happen. Not until I click the "Next difference" button does TC 9.12 start counting differencies and the progress bar grows - in this state the "Abort" button works - and the count takes approximately the same 2 minutes as in TC 8.52a.

TC 8.52a seems more responsive than TC 9.12 when navigating op and down in the CbC window with these large files - and one time when navigating TC 9.12 crashed.

When comparing some smaller zip files of 950 Mb TC 9.12 works like 8.52a - % count and progress bar works - maybe TC 8.52a is a little more responsive.

I have tested some of the previous TC 9x versions - and the problem is there already from TC 9.0b1.

The computer this is tested on is an Intel Core i7-4790 CPU 3.60GHz with 8 GB ram (6.5 Mb free). OS is Windows 7 x64 Pro
License #524
Danish Total Commander Translator
TC 9.21a 32+64bit on Win XP 32bit, Win 7, 8.1 & 10 64bit and TC 2.92b1 on Android 6.0
Get: Extended Total Commander Menus | PHSM-Calendar

umbra
Power Member
Power Member
Posts: 783
Joined: 2012-01-14, 20:41 UTC

Post by *umbra » 2018-01-14, 02:18 UTC

My test with two completely different 10GB files, run on Win10 x64, 32GB RAM, SSD, Haswell i7

TC 8.52 x64
The progress bar and percentage counter appear immediately, it takes them about 40s to reach 100%, stay there for 5s and then TC shows number of differences. WorkingSet is by ~11.5GB bigger than PrivateBytes.

TC 9.12 x64
The progress bar and percentage counter appear immediately, it takes them about 10s to reach 100%, stay there for 5s and then both reset back to 0%.
The dialog window responds to mouse, scroll bars and jumping between differences works as well, but it cannot be closed, the number of differences is not shown and TC keeps utilizing one CPU core. WorkingSet is by 4192MB bigger than PrivateBytes. The active thread is
KERNELBASE.dll!VirtualQuery+0x28
TOTALCMD64.EXE+0x19c27

Like petermad, I did not observe any crash. And I also confirm it can be reproduced since 9.0b1.
Windows 7 Pro x64, Windows 8.1 Pro x64, Windows 10 Pro x64

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 36554
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) » 2018-05-04, 07:41 UTC

This should be fixed in TC 9.20 beta 1, please test it!

The error only occurs when the first difference is near the end of the file.
Author of Total Commander
http://www.ghisler.com

User avatar
petermad
Power Member
Power Member
Posts: 8073
Joined: 2003-02-05, 20:24 UTC
Location: Valsted, Denmark
Contact:

Post by *petermad » 2018-05-04, 15:49 UTC

This seems to be fixed in TC9.20b1 - but there is still an issue:

If I press the Abort button during the counting of the differencies, and then press Compare, the panels are cleared, it says "Press ESC to abort" in the lower left corner, but nothing happens for 1½ minute, then the panels is filled up and the counting starts. It would be nice if in that period the cursor went to Wait state, or a new loading dialog was shown, or it just showed "Loading files" in the lower left corner in stead of "Press ESC to abort" - to indicate that TC is actually working.

The behaviour is now the same as in TC 8.52a

Testet with two 11 Gb zip files where there is added on file extra to one of them.
License #524
Danish Total Commander Translator
TC 9.21a 32+64bit on Win XP 32bit, Win 7, 8.1 & 10 64bit and TC 2.92b1 on Android 6.0
Get: Extended Total Commander Menus | PHSM-Calendar

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 36554
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) » 2018-05-06, 08:56 UTC

What happens is that TC recompares the entire files - that's the same comparison which happens in the progress dialog in Total Commander itself when you compare the files from there.

I will add a progress bar in the compare window for this step too.
Author of Total Commander
http://www.ghisler.com

User avatar
petermad
Power Member
Power Member
Posts: 8073
Joined: 2003-02-05, 20:24 UTC
Location: Valsted, Denmark
Contact:

Post by *petermad » 2018-05-09, 19:54 UTC

I will add a progress bar in the compare window for this step too.
Confirmed in TC 9.20b2 :-)
License #524
Danish Total Commander Translator
TC 9.21a 32+64bit on Win XP 32bit, Win 7, 8.1 & 10 64bit and TC 2.92b1 on Android 6.0
Get: Extended Total Commander Menus | PHSM-Calendar

Post Reply