Compare by content file size limit

Please report only one bug per message!

Moderators: Stefan2, white, sheep, Hacker

Post Reply
likeasomebooody
New Member
New Member
Posts: 1
Joined: 2019-04-07, 02:20 UTC

Compare by content file size limit

Post by *likeasomebooody » 2019-04-07, 02:42 UTC

I use compare by content a lot, it's a great feature. Recently I tried to compare close to 4TB worth of content. There were several massive binary files ranging from 80GB to 1.1TB in size. When copying that much data I typically take the time to verify no data corruption due to subtle hardware/RAM failure took place. I used version 9.0a later updating to 9.22a. Under "synchronize directories" dialog asymetric/subdirs/by content/ignore date check boxes were ticked. Both versions failed to do a by content comparison simply skipping the content compare. There was no error message or warning of any kind. I tried several times on two different machines and in all cases comparison done as if "by content" checkbox was not checked. Both source and destination were USB drives connected to USB 3.0 hub copying at USB 3.0 speeds. I used windows 8.1 and windows 10 LTSB.

What is the maximum file size that TC compare by content feature can handle? And if the limitation exists is it possible to remove it?

User avatar
Usher
Senior Member
Senior Member
Posts: 402
Joined: 2011-03-11, 10:11 UTC

Re: Compare by content file size limit

Post by *Usher » 2019-04-07, 23:08 UTC

TC must buffer comparing results before displaying them. Do you think that TC can use the whole available space in RAM and on disks (maybe another 4 TB) for that job? Do you realize how long it will take?

Do you really need to know location and value of any different byte? What is wrong with using "synchronize directories" or "find duplicate" options to eliminate duplicates and limit number of files for comparison?

I can't see any bug here, it's rather missing feature - TC might warn that it was mission impossible.
Regards from Poland
Andrzej P. Wozniak

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 37417
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Compare by content file size limit

Post by *ghisler(Author) » 2019-04-08, 17:04 UTC

TC doesn't buffer them by itself, it just maps the two files as a whole into memory.
Author of Total Commander
http://www.ghisler.com

User avatar
Usher
Senior Member
Senior Member
Posts: 402
Joined: 2011-03-11, 10:11 UTC

Re: Compare by content file size limit

Post by *Usher » 2019-04-08, 19:57 UTC

2ghisler(Author)
Do you mean virtual memory?
What about comparing two BluRay disk images?
Regards from Poland
Andrzej P. Wozniak

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 37417
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Compare by content file size limit

Post by *ghisler(Author) » 2019-04-11, 10:46 UTC

Author of Total Commander
http://www.ghisler.com

User avatar
MVV
Power Member
Power Member
Posts: 8276
Joined: 2008-08-03, 12:51 UTC
Location: Russian Federation

Re: Compare by content file size limit

Post by *MVV » 2019-04-11, 10:58 UTC

Perhaps it would greatly reduce memory requirement if TC would limit mapping window size (e.g. by an INI option) and move it during comparison. It should be easy to do for just comparison but it be a bit harder for visualization, since it will require more window movements. Text mode may be disabled for files that don't fit into configured window.

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 37417
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Compare by content file size limit

Post by *ghisler(Author) » 2019-04-15, 13:56 UTC

It would make the comparison and display much more complex. TC would constandly need to check whether the position goes beyond the current buffer, and also subtract the buffer offset from the address.
Author of Total Commander
http://www.ghisler.com

User avatar
Usher
Senior Member
Senior Member
Posts: 402
Joined: 2011-03-11, 10:11 UTC

Re: Compare by content file size limit

Post by *Usher » 2019-04-15, 15:38 UTC

2ghisler(Author)
So just add warning if file is too big to compare, please…
Regards from Poland
Andrzej P. Wozniak

User avatar
DrShark
Power Member
Power Member
Posts: 1264
Joined: 2006-11-03, 22:26 UTC
Location: Kyiv, 68/262

Re: Compare by content file size limit

Post by *DrShark » 2019-04-15, 16:56 UTC

ghisler(Author) wrote:
2019-04-15, 13:56 UTC
It would make the comparison and display much more complex. TC would constandly need to check whether the position goes beyond the current buffer, and also subtract the buffer offset from the address.
If it will make it possible to compare larger files on 32-bit Windows, I think it's worth to consider for implementing.
Android 4.3.1 no root, kernel 08.09.2016; Vista Home Premium SP2 rus 32 bit
TC #149847 Personal licence

Cuz we're all in this together, We're here to make it right

User avatar
Usher
Senior Member
Senior Member
Posts: 402
Joined: 2011-03-11, 10:11 UTC

Re: Compare by content file size limit

Post by *Usher » 2019-04-15, 17:34 UTC

DrShark wrote:
2019-04-15, 16:56 UTC
If it will make it possible to compare larger files on 32-bit Windows, I think it's worth to consider for implementing.
Yes, but still with warning that it will take a long time and with setting for max file size.
Regards from Poland
Andrzej P. Wozniak

User avatar
MVV
Power Member
Power Member
Posts: 8276
Joined: 2008-08-03, 12:51 UTC
Location: Russian Federation

Re: Compare by content file size limit

Post by *MVV » 2019-04-16, 05:05 UTC

ghisler(Author),
Yes, it is harder to implement, but it is the right way.

At least adding a question before loading large files into compare dialog would be nice.

Currently we have:
1. Compare two large files in TC,
2. TC sees a difference, so compare required.
3. TC opens compare dialog.
4. TC shows an error:

Code: Select all

Not enough memory, search aborted.

The two files are DIFFERENT!
Would be nice to get the following:
1. Compare two large files in TC,
2. TC sees a difference, so compare required.
3. TC checks file sizes and, if one of the two files is large, asks user:

Code: Select all

The two files are DIFFERENT!
Do you want to see differences between two large files?

C:\file1.bin
C:\file2.bin
4. TC opens compare dialog.
5. TC shows an error: "Not enough memory, search aborted. The two files are DIFFERENT!"

Post Reply