Russian letter "ya" in Lister

[ghisler(Author)]

Sorry, I really don't know why it works fine here. I did change the language for non-Unicode programs to Russian too...

[DrShark]

Lister had many changes since this issue has been reported (btw, it was first time mentioned in 2006 in Russian TC mailing list), it's interesting to know whether the problem still exists. I cannot reproduce it on Vista in Lister from TC 7.04a, or even old standalone 32-bit Lister from 2005, but all users who confirmed have been using Windows XP or 2003 then...

[Flint]

I could reproduce it in a virtual machine with WinXP SP3 rus, TC versions 9.50b5, 8.51; but it works fine in Win7 rus, TC versions 9.50b5, 8.51, 7.50a, 7.04a. So I guess, it has something to do with the operating system.

[DrShark]

Christian Ghisler finally reproduced the problem on his Windows XP. In short, it seems direct fix for this is impossible.

Detailed explanation (from email):

Apparently it's a bug in the built-in Windows function TabbedTextOut. It only seems to occur under these conditions:
1. There is a tab after the Ya character
2. Uniscribe is not used. Total Commander uses Uniscribe under one of these conditions
- Mode Unicode or UTF-8
- Ansi with Hebrew or Arab encoding
- Ansi if the font is set to something other than "default"

So it displays correctly here when setting the encoding of the font to ANSI or Cyrillic Windows (1251), because they both use Uniscribe.
With default encoding, Total Commander just calls TabbedTextOut with the unchanged line.

[...] the only solution would be to use Uniscribe, but this could have other consequences like different font substitutions used [...] (UniScribe uses different font substitution than regular GDI string functions when a string contains a character which isn't part of the current font.) [...] etc.

I can add here that a problem with Russian Ya (я) letter cut seems to happen only in Lister's "Text only" view, and only with "Encoding"->"As configured by current Font" set as encoding which disables Uniscribe by default.

It matters because in addition to above workaround (to use Uniscribe by choosing "Encoding"->"ANSI (local code page) (0)" or "Cyrillic Windows (1251)"), since Total Commander 9.51 RC3 after setting wincmd.ini [Lister] Uniscribe=2 Lister uses Uniscribe even with "Encoding"->"As configured by current Font" option active. Uniscribe is not set to 2 by default because of mentioned above possible "different font substitutions used etc." problems.

And indeed, though Uniscribe=2 fixes the cut of "я" letter in "Text only" view with "As configured by current Font", but on Windows XP it introduces character substitution issue at least with Binary, Hex and Unicode (6) views.
On below screenshot, an illustration of control characters (Tabs, Enter) substitution in Lister's Binary view in a test file which was used to reproduce "я" letter cut problem:
https://live.staticflickr.com/65535/49675725762_eb38a043c2_z.jpg

• on middle window Lister uses dots as substitute characters, which on XP happens if "ANSI (local code page) (0)" or "Cyrillic Windows (1251)" is used, and in this case it doesn't matter whether Uniscribe=0 or 2;

• on left window Lister uses "As configured by current Font" option with Uniscribe=0, and substitute characters aren't dots;

• on right window Lister uses "As configured by current Font" option with Uniscribe=2, and substitute characters are invisible.

On Vista Lister always uses dots as substitution characters no matter what combinations of above options are set.

Christian Ghisler wrote that it seems there is no way to change above characters substitution behavior on XP when "As configured by current Font" is active:

[... with "As configured by current Font" option the] font is created - with [...] codepage set [...] to "default" (it's when you create the font with parameters DEFAULT_CHARSET, ANSI_CHARSET or RUSSIAN_CHARSET)

Lister's Options menu have a checkmark on "ANSI (Windows charset)" option, but "As configured by current Font" menu item itself doesn't indicate what exactly this "default" encoding is, so maybe it's a kind of CreateFont function bug on XP so it uses characters different than dots for substitution if font doesn't not have codepage set in explicit way.