[FULLY FIXED!] Bugs with rendering HEX view of a file

[AntonyD]

Have you tried using UTF-8 mode instead of code page 65001?

It seems we understand some aspects of using the program differently. Yes, all modes can basically be enabled when viewing. They will all work. They will show something—what they are currently designed for. So yes—if I enable mode 7 - UTF-8 - then the readable text contained in this file will be displayed in the correct readable Russian encoding. BUT the situation is not quite that simple. When we open a binary file, your algorithm ALREADY and ALWAYS correctly enables mode 2 - binary! The program itself understood WHAT needs to be shown and how.
And this strict "tabular" form of text representation is quite convenient - but only when this text is within the ANSI encoding ))

+ [Encoding] menu - "As configured for current Font" option is toggling in for this case. And this is also true for most cases! So here I immediately and clearly see that the open file is not such simple as it could be, and even if I need to find something in it, the first thing I now have to do is figure out its encoding. And therefore, LOGICALLY reasoning, I go to the menu responsible for this!!! And I enable 65001 in it. And basically, I see confirmation of my thoughts – I begin to understand and read the text that can be seen in the binary file in an open form.

It is illogical for me to enable the PURE UTF-8 mode, which is intended for a simple, completely text-based file. In my logic. At least because the text spreads outwards - having no restrictions. And therefore, if I suddenly see that the binary mode with encoding 65001 does NOT show what is needed, I just report it as a clear error. But I also clearly do not expect the solution to be a suggestion to 'just enable another mode, and forget about this one - it's not fully correct'.

If we acknowledge the situation as erroneous, then even if fixing it takes a lot of time—and we will have to leave it in the plans for the next version, not for the upcoming beta—let's just agree on that. Will fix it for v12.
But let's not consider the suggestion to switch modes as a fix for this problem.

I don't have this problem. Can you give an example with the file exif_ru.lng? What do I need to select, and in which mode, to get wrong copied text?

But let's forget about this problem for now, because sometimes I manage to repeat it, and sometimes I don't. So I will wait until I have a 100% understanding of how to repeat it. In the meantime, let's think that there is no this problem))))))

[ghisler(Author)]

This should be fixed now in Total Commander 11.57 RC1, please check!

I have added this broken character handling from hex view to binary view now in Total Commander 11.57 RC1!. It required quite a lot of changes, so there will probably be new bugs. Please test it, and report any new problems here.

23.03.26 Fixed: Lister: When scrolling through lines with cursor up/down, try to move sideways as little as possible when characters have variable width (32/64)
23.03.26 Fixed: Lister: Binary view incorrect when code page for non-Unicode programs in Windows was set to UTF-8, or the code page in Lister was set to 65001 (32/64)

[AntonyD]

Sort of on my files, where everything is already very visual "stuck in my head" for me and any inconsistency will be visible momentary -
it all looks very fine.
Especially surprised that the selection made in mode 2 - when switching to mode 3 remains! I mean, if I find the text I want, but, uh...
I suddenly need or need exactly byte representation of it - I just switch to mode 3 and copy the codes)))))) - A delightful solution!

P.S. \\
The only thing I've found is that when I switch between modes like above (2-3-2-3...) - there is an automatic and unwanted scrolling down
of the file contents. In both files but for mode 3 scrolling length is more than on mode 2.

[white]

Pls, load this file https://www.upload.ee/files/18660981/exif_ru.lng.html
Open with Lister in a mode #3, set encoding to 65001 (UTF-8)

I used TC 11.57rc1 32-bit and 64-bit for testing.

* Open the file in Lister, in mode #3, encoding set to 65001.
* Make sure showing the cursor is enabled.
* Place cursor at the beginning of "изображ" in the beginning of the file.
* Move the cursor to the right by pressing arrow right repeatedly.
* When the cursor it on "ж", it jumps to the next line, not after "ж" on the same line.
* But when you then press the arrow left key, the cursor jumps to the position after "ж". This is a position you can not reach by clicking with the mouse.
* Now, press Shift+ arrow left. The "ж" character is selected, but only one hex value is selected. You cannot do this by selecting the character using the mouse. This same issue also happens if 2 bytes character is on one row, for example with the character "в" a little lower.

[AntonyD]

You've described it the way it is. But there is a nuance - I don't see the described steps as a bug(-s) at all -
it's just a fact of how Lister works in cursor mode. In current variant of its code....
The cursor - Which by the way I never turn on in this mode - it's a definite wasteful action.
So if any of the described moments you consider a bug - then specify in the description.
Which item did you dislike the most?

I'm just getting used to HOW Lister shows HEX codes now. But ideally I dream of showing data in a fully fixed tabular output form.
I already sent an example to Mr.Ghisler, by the way! dear author! - could you attach a compiled version of my program here?
You know, the one where I show spaces for double-byte characters? You've tested it to see how it works, so you've already built it.
I just don't have access to a computer right now, where I complied it.
Yep and for some reason I got it for 20+ megabytes - probably everything was statically linked inside..... not a good example of
a final exe IMHO(((
then we can get at least a few more independent opinions on what and how it should be rendered in this mode and
how the selection should work.

[ghisler(Author)]

The only thing I've found is that when I switch between modes like above (2-3-2-3...) - there is an automatic and unwanted scrolling down of the file contents.

since the line breaks are different, it's not possible to go to the exact same position in all modes. For example, the first line in hex view may be in the middle of a line in binary view.

When the cursor it on "ж", it jumps to the next line, not after "ж" on the same line.

Unfortunately I couldn't find a better solution here. The problem is that the "ж" stretches across both lines - the first byte is at the end of the first line and the second byte at the start of the second line. So the cursor being after the second byte is essentially on the second line.

But when you then press the arrow left key, the cursor jumps to the position after "ж". This is a position you can not reach by clicking with the mouse.

This is the compromise I used - the cursor goes back just one byte here, or half a character. When you then select with shift+cursor left/right, it will select just one byte, which also reflects in the copied text (Ctrl+C).

[AntonyD]

since the line breaks are different, it's not possible to go to the exact same position in all modes. For example, the first line in hex view may be in the middle of a line in binary view.

I understand that, but there seems to be a whole different emphasis of inappropriate behavior I'm describing here.

Scrolling IN PRINCIPLE is happening, even though I'm just repeating the same action!
It seemed to me that for ANY internal mode(w/o plugins) switching it would be adequate to store as a cache of some kind of generalized
information about the content. And specifically, that when we leave Mode 3, then its state is temporarily memorized - as is the selection itself
in particular - including the position of the thumb on the scroll bar. That is, when we return to this mode, in addition to restoring
a selection, we also restore the position of the thumb on the scroll bar. So we don't calculate again - we REMEMBER this position - and when
it becomes needed we are simply remembering it.

[white]

When the cursor it on "ж", it jumps to the next line, not after "ж" on the same line.

Unfortunately I couldn't find a better solution here. The problem is that the "ж" stretches across both lines - the first byte is at the end of the first line and the second byte at the start of the second line. So the cursor being after the second byte is essentially on the second line.

This makes sense to me. It's no different from ANSI characters. There are no newlines in hex view.

But when you then press the arrow left key, the cursor jumps to the position after "ж". This is a position you can not reach by clicking with the mouse.

This is the compromise I used - the cursor goes back just one byte here, or half a character. When you then select with shift+cursor left/right, it will select just one byte, which also reflects in the copied text (Ctrl+C).

This makes no sense to me. In my opinion the behavior should be consistent. Cursor movement using arrow left/right should be consistent, moving the cursor to same positions in either direction. With the mouse when selecting characters (at the right side), you cannot select "half" characters. Using the keyboard you can only select a half character when it is at the end of a line. That's not consistent. And when moving the cursor backwards and positioning the cursor behind "в" (first occurrence in the file) and then pressing Shift+arrow left, half the character is selected while both hex codes are on the same line. This is not intuitive at all. So in my opinion a cursor position at the end of a line should never be allowed.

[AntonyD]

Well, if you ask my opinion - the cursor functionality for this view mode should be disabled and unavailable at all.
This mode - like mode 2 - are very special variants of text manipulation - where the presence of a cursor only complicates the usual behavior.

[ghisler(Author)]

Well, if you ask my opinion - the cursor functionality for this view mode should be disabled and unavailable at all.

I strongly disagree, it's useful to select and copy text from binary files.

[AntonyD]

Well, for example, white just above pointed out the illogical nature of the current behavior when working ONLY with the keyboard.
Isn't enough?
Although... is it really that convenient to use it? When you are in a very narrow column of data, your eyes can only cover a not very significant
amount of data – and therefore selecting with the mouse becomes the easiest, considering the fact that the cursor blinks and does not show its
position accurately and consistently everywhere – but I don't even create a bug for that anymore))))

But if you can just fix what is mentioned above, well, that means the app will simply work better for everyone)))

[ghisler(Author)]

Better a bit illogical than no cursor at all!

[AntonyD]

Well, I basically also wrote above that I don't consider everything described as a bug, either in general or in particular. The main thing is that the display on the right borders of symbols rendering has been fixed and the selection in general works as it should. In this version, this topic can probably be considered closed as "fixed".

[ghisler(Author)]

I think I will still try to select entire characters/skip over entire characters when the user presses Shift+Cursor left / Cursor left, respectively, when at the start of the line. This way it will only be possible to select partial characters when selecting the hex part with the mouse.

[white]

29.03.26 Fixed: Lister, binary/hex view with codepage 65001: on Cursor left, skip over entire multi-byte characters also when they stretch across 2 lines (32/64)

Seems to work OK, except when pressing the End key, for example on the first line with the character "в" in test file exif_ru.lng.

Another issue:
When an character skips over to the last line of the file, a ghost character is displayed at the end.
I truncated the test file at the beginning:

Code: Select all

00000000: 45 58 49 46 4C 4E 47 00|72 75 01 01 00 00 00 01  EXIFLNG ru..   .
00000010: 00 00 24 00 00 00 D0 A8|D0 B8 D1 80 D0 B8 D0 BD    $   Ширин
00000020: D0 B0 20 D0 B8 D0 B7 D0|BE D0 B1 D1 80 D0 B0 D0  а изображ
00000030: B6                                               е