[FULLY FIXED!] Bugs with rendering HEX view of a file

[AntonyD]

it's pretty weird. considering all I do is write about the same thing all the time.
Let's start with the simple stuff - were you able to build and run my 2 programs?
Were you able to output a HEX view of the original test file using them?

[ghisler(Author)]

Yes I have, but it doesn't seem to work for me. I get output like this with Lazarus and UTF-8-encoded exif_ru.lng:

Code: Select all

          0  1  2  3  4  5  6  7   8  9  A  B  C  D  E  F  | 0123456789ABCDEF
         --------------------------------------------------|-----------------
00000000  45 58 49 46 4C 4E 47 00  72 75 01 01 00 00 00 01 | EXIFLNG.ru......
00000010  00 00 24 00 00 00 D0 A8  D0 B8 D1 80 D0 B8 D0 BD | ..$... ? ? ? ? ?
00000020  D0 B0 20 D0 B8 D0 B7 D0  BE D0 B1 D1 80 D0 B0 D0 |  ?  ? ? ? ? ? ? 
00000030  B6 D0 B5 D0 BD D0 B8 D1  8F 00 00 00 00 00 01 01 | ? ? ? ? ?.......
[...]

[AntonyD]

This was clearly only one of the two programs. You tried the console one, and it is quite logically tied to how
your console is configured + the locale. And that is quite acceptable in our conditions. Bad result... yeah...
I am, on the other hand, doing great – I showed a picture of how this program worked for me...
https://i.ibb.co/wZSLKqyB/2026-03-04-125437.png
And at the end, instead of the RU-words "Нажмите Enter...", it just showed the question symbols? Isn't it?
Most likely, you really need to first set the RU locale in the system, and then you will get the correct
output in the console as well.

And now the second program is with GUI - what about it - what result do you have there?
For this program, I do NOT expect that the normal text rendering on a right side will not work.
ALTHOUGH, of course, we can leave *some* doubt on the lack of the necessary fonts, but you
seem to have already checked some bugs - where the output was in Russian - and you had it.
That is, you can basically get it on your computer.

And I'll tell you right away what the essence of the second program is - it shows that, first of all, the output of a void in front of
two-byte characters is possible. Separate allocation of bytes of emptiness and bytes of a symbol is also possible.
The display of "broken" sequences, where one byte is on one line, the second on another, is also possible without unnecessary characters.
And it's definitely been tested on my laptop. IF it all works the same way on yours, then something is wrong in the current Lister code.
with the processing of these sequences, with possibly additional conversion, which occurs in the depths of its logic. Or somewhere else, but
it clearly depends on some external factors that will need to be found together somehow.

[ghisler(Author)]

I hadn't tried that yet, but it seems to work better. It's a totally different approach than mine, in function DecodeLineToChars:
I look for the first primary UTF-8 character and then convert all the bytes of the line directly to UTF-16. You, on the other hand, convert primary characters to the UTF-16 equivalent of the entire UTF-8 sequence, and secondary characters to spaces.

Both have their advantages and disadvantages: Mine is better readable for humans because there are no extra spaces, and yours provides a 1:1 mapping of bytes to characters.

[AntonyD]

In particular, this is the effect I was aiming for - because viewing in HEX codes can in no way be called an everyday thing that a person just needs
for some casual "here and now" viewing. It is always a special control and verification of WHAT is in a particular file - literally WHAT is in a file.
And if we just collapse all the bytes into a purely human string - it loses that control and validation.

After all, if we just need to copy text that is understandable to a human, then in my program's context menu (you've seen this feature, right? Have you tried it?) we can ultimately get this text in an understandable and familiar form. So the task is solved in any case. BUT my version with guaranteed accuracy solves the edge cases of symbol representation at the boundaries of the 'HEX table.' Unfortunately, I am currently observing this issue in the current implementation of Lister's code, when a single letter 'ж' gives me two non-printable characters on the screen rendering...

So in the end - what do you think - can we come to some reasonable compromise and solution for high-quality rendering of hex codes of a binary file? If anything, you can use my code however you like)))) It's not copyrighted))))

[ghisler(Author)]

I don't know whether your approach will work for all languages. For example, there are full width Katakana and Hiragana and half width in Japanese, where the full width take as much space as Chinese ideograms, and half width as much as English letters. And then there are ways to write an accented letter by a letter followed by an overtyping accent, which is mainly used by MacOS.

[AntonyD]

I don't know this either – because in the overwhelming majority of cases only native speakers will be able to carry out all the checks
to understand and evaluate what has been produced. Therefore, even if something is changed, it should simply be done as a setting –
like this you can open a file (as it has always been and what everyone is used to) – or like this (and please check your file/encoding options).

Well, then we still need to somehow understand why the letter "ж" can still be drawn clearly with a single character on your side(comp?),
but on my side it can't be done. And moreover, not only in your program — I found a couple of others where the same problems exist.
BUT in those, they authors definitely won't be able to fix anything.

[ghisler(Author)]

Have you tried with a different font?

[AntonyD]

If the question concerns checking Lister's code – then yes, and I have already written about this – this error exists in any allowed font. I have also already written that I found a couple of other programs that were written long ago and are no longer updated, but contain exactly the same characteristic error of displaying a readable representation of multibyte sequences in HEX format. That is, this is not just your error – it is something integral, since those programs were also written in Pascal. Well, their authors will definitely change nothing anymore. Their time is over...
Image: https://i.ibb.co/3VQDsmx/2026-03-10-171732.png Just for example.

[ghisler(Author)]

I have changed the language for non-Unicode programs to Russian, but I still can't reproduce the error with the "ж" character. Lister is directly using Windows functions to render the text, so I don't think it has anything to do with Pascal.
There are two things you can try:
1. It's possible that the "ж" character has the wrong character width. Then TC replaces it by default. To avoid this, add the line
ReplaceWrongWidthChars=0
to wincmd.ini section [Lister].
2. Lister uses Uniscribe to render the text by default. To disable Uniscribe, add the line
Uniscribe=0
to wincmd.ini section [Lister].

[AntonyD]

I have changed the language for non-Unicode programs to Russian

But that's not obviously enough) You should SET the whole lang of OS - to RU. ALL OS environment should became Russian.

Plus Did you read the part of my comment where I write that I found a similar program, also written in Pascal, and it has exactly the same error!?
I even attached a photo! So it's clearly evident that you are using some single piece of code that doesn't always work the way it should.
As for the mentioned variables, ReplaceWrongWidthChars=0 literally changes nothing.
Uniscribe=0 - this is clearly the answer to the question of how you draw text. In my version of the code, in my program, it was clearly not enabled. And everything works as it should. In your case, you enabled some internal?|hidden? Windows mechanism that introduces its own errors and glitches. As soon as I turned it off, I completely lost the ability to display text in a readable form in UTF-8 encoding when also we use HEX mode as well.
It feels like these states have become completely non-intersecting. Lister won't be able to do one thing if you want to do another along with it.
The best option I can get is: Image: https://i.ibb.co/3y2R47R0/2026-03-11-124911.png
And pay attention to the symbol that I point with the arrow. IT's always like this! And even for the case when I'm trying to get
a readable UTF-8 representation of the entire text, that's what I get: Image: https://i.ibb.co/xVzHRKv/2026-03-11-125403.png
And as you can see, it has the same symbol in the same place again - ALTHOUGH all the other symbols seem to have changed, although
not in the form that we expected to see.

[ghisler(Author)]

So the character shows up when you set Uniscribe=0? That's odd, Uniscribe works absolutely fine here. It's used to draw combined left to right and right to left text correctly, especially when selections are involved, so I will not turn it off for everyone.

[AntonyD]

Well even for me it doesn't have to be turned off - because without this mechanism I can't see anything useful in this file viewing mode at all.
With it so far there is only one mistake)))))

But did you understand - WHAT did I show you even when this mechanism was on? The second byte from UTF-8 is drawn by a symbol similar
to the end of the paragraph. I marked it with an arrow And this symbol is ALWAYS drawn.
Thus the feeling that the drawing, regardless of the use of Uniscribe, goes in a symbolic version 1-by-1 (not a string in a whole).
And just by turning on this mechanism you still manage to draw other characters more correctly from human point of view.
BUT somehow in the symbolic version you are different from the same symbolic rendering that I made in my program.

[ghisler(Author)]

Interestingly, I CAN reproduce the problem when setting the encoding for non-Unicode programs to Russian BUT with the option to use UTF-8 encoding. When I use this, my new code to correctly align the characters isn't used. I will update my code, maybe it also helps in your case.

[AntonyD]

I'll go further - I have tried quite a lot of programs for displaying the HEX codes of files, and almost all of them either had no support for displaying UTF-8 encoding at all, or the error w/ "ж" was the same as I showed above in several examples. Therefore, this very feature of being able to actually change the encoding and display it alongside the HEX codes is very useful and in demand. It would be great if all sorts of empty whitespace characters for multibyte encoding were also rendered - so that you can still see the correspondence between the code and its character representation more logically - roughly as I managed in my program))))))))

But in any case, many thanks for the support in this direction, and quite possibly a relevant fix )) I believe it will work.