[x64] Compare by content and big files

[Sob]

This was known limitation in 32-bit TC, but I believe it's bug in 64-bit TC. As explained in other threads, when showing differences, TC maps files in memory. And it's not the matter of actual free memory, just the address space, which is quite small in 32-bit mode. So this was one of the reasons why I was looking forward to 64-bit TC.

Unfortunately nothing changed. When comparing two 2.6GB files, TC shows the same old "Not enough memory, search aborted".

Win7 x64, TC 8.0 x64 beta 8.

[ghisler(Author)]

Currently the function is limited in code to 2 GB for each file. You should already be able to compare two 2GB files, something which was impossible with TC 32-bit. Maybe I will add support for files > 2GB in a later version, but it will require many changes.

[Sob]

Ok, better than nothing. It's not like I'm hitting 2GB limit very often. It was much worse with 32-bit TC where even few hundered MBs files often caused problems.

[ghisler(Author)]

I have enabled it now, but there is no guarantee that it will work correctly. So far I could only try it with binary files > 2 GB (max 6 GB).

[Flint]

Quick test in 8.0β9 revealed no problems. I tried to compare two binary files with sizes 4,6 Gb and 3,3 Gb; Win7 x64, 16 Gb RAM.

[ghisler(Author)]

Thanks for your feedback! It would be great if some other users could test that too!

[HolgerK]

(Win7 x64 12GB physical RAM)
Successfully tested with 2 different files: 4,295,032,700 Bytes <-> 4,294,967,340 Bytes
and 2 identical size (but slightly different content) files : 6,901,953,052 Bytes

A progress dialog would be nice... currently the increasing memory curve of task manager's performance tab is the only indicator that something is going on.

Regards
Holger

_________________
First time since 18 months that i've got the feeling that the amount of installed memory could be more...

[Sob]

TC 8.0 beta 9 x64, Windows 7 x64, 8GB RAM, two 10GB files -> no problem, works great.

Well, it was painfully slow, but it's because of slow HDD and no chance to cache 20GB in ~6GB free RAM. :)

I agree with progress dialog idea, it would be useful even with much smaller files. Two 100MB files sitting on the different ends of same physical disk are enough to freeze TC for few seconds. It's no real problem to live with it, it just doesn't look very nice.

[ghisler(Author)]

Thanks for your tests! Were these all binary files, or also some text files (e.g. large log files)?

I will check whether I can show some kind of progress bar.

[Sob]

I now tried it with two 6GB txt files (almost the same, with only few differences) and no problem.

[HolgerK]

Tested with two text files 3,930,107,033 and 3,930,107,034 Bytes.
Sucessfully edited one file to make it the same like the other.
Writing the edited content was fast, but the time until the compare window shows a content was really long (>1h).

Regards
Holger

[ghisler(Author)]

Thanks for your tests! I know, the compare function isn't really suited for such large files...

[HolgerK]

15.11.11 Added: Compare by content: Show compare progress bar and percent done in footer (32/64)

Sometimes this does not work.
After a different amount of percentage(5%-10% in my test case with >3GB files) the compare contents dialog's window caption bar becomes a "(Not responding)" and the progress bar stops updating.
The compare itself is okay and after a while the compare window shows the content of the two files.
Interesting the <Esc> key still works even if the windows shows "(Not responding)".

If i switch (after the first binary compare) to [_] binary and press [Compare] again, the dialog is always responding(even dialog resize was possible).

15.11.11 Fixed: Compare by content: Increase load speed (slowdown caused by try..except block) (64)

I do not have absolute values, but the compare progress seems to be a lot faster, especially if comparing a second time (from cache).

Thanks
Holger

[ghisler(Author)]

After a different amount of percentage(5%-10% in my test case with >3GB files) the compare contents dialog's window caption bar becomes a "(Not responding)" and the progress bar stops updating.

Hmm, cannot reproduce. What are you comparing, binary or text? And how big are the files?

[Sob]

It happens here too. Binary compare only, even with relatively small files and it's very predictable.

When comparing files for the first time (they are not in Windows disk cache):
- for 180MB files TC stops responding around 35%, for 750MB files around 9%, so it seems it's always after reading the same amount of data, around 65MB
- TC's CPU usage is low for both before and after it stops responding

Second compare of cached files:
- does not stop responding for 180MB ones, stops around 55% for 750MB ones
- TC's CPU usage is 100% of one core for whole time

System is Windows 7 x64, 8GB RAM (6GB free) and not very fast disk. Test files are completely different, only have similar size, e.g. two different iso images.