Lister: default font (for Unicode/UTF-8)

[rsa]

The integrated Lister uses old fonts as default fonts: "Fixedsys" for ANSI and "Terminal" for ASCII.
The default font for variable width is "System" (bold). This font only exists as bold and bold-italic.

First I'd like to say, that all these three fonts are quite old and IMHO not very nice. The font "System" is not available in a standard font (not bold, not italic).
These old fonts also have shortcomings on common Unicode characters, e. g. the en dash ("Halbgeviertstrich", Unicode code point U+2013, UTF-8 hex e2 80 93).

On the one side it's okay, if the font for ASCII only supports the characters of the 7 bit ASCII Code and respectively the font for ANSI supports all 256 characters of the 8 bit Code. However, the used code page is another point which has to be noticed.

On the other side there is the behavior on Unicode-files. If a file is encoded in UTF-8 (and has a Byte Order Mark for UTF-8), Lister (with default "Autodetect") automatically activates the display option "UTF-8" and ANSI. So the font Fixedsys is used, but this font hat no full support of Unicode; it also does not support widely used characters, like the en dash.
If the default font would be a font with better Unicode support, like Consolas, also the en dash will be displayed correct.

Another proper way might be, to change the Lister and define also a font for Unicode (all UTF-formats), where Consolas could be a good option as default font. The shortcut (key) "U" might be a good way for this. So it's possible to have a proper Unicode font for monospace (default) and a way to switch to a proportional font with key "V".
The good default font for variable width might be "Segoe UI".

Consolas is the default monospace font in Windows, since Windows Vista, i.e. since 2007!
I think, it's time to use this font also for the Lister as default font.


Of course, everyone can configure his own fonts, but there seems to be a problem (?) with the font size.
If I chose Consolas, Standard, Size 11 in the config dialog "Schriftart", it will be displayed as "Consolas,-16" in config dialog "Configure Lister". It's the same of all fonts and all sizes: the used sized is bigger than the selected size. For me this is a little bit strange.



BTW: The name of the Lister-option "Unicode" (#6) is IMHO not the best.
Also the help text ("Unicode ist ein Dateiformat, bei welchem jedes Zeichen 2 Bytes benötigt.") is a little bit imprecise. Unicode is a standard for encoding, representation, and handling of text, which an be implemented in different encodings, e.g. UTF-8, UTF-16, UTF-32 and others, like the UCS forms. Please also note, that the are also code points (characters), which need two 16-bit code units!


P.S.: Windows 10 Pro (64 Bit, DE) with Total Commander 9.12 (64 Bit).

[ghisler(Author)]

Sounds like a good idea to use Consolas for fixed pitch fonts, if it is installed. I will check whether I can add it.

[browny]

BTW: The name of the Lister-option "Unicode" (#6) is IMHO not the best.

Following the logic of the next menu item UTF-8 (#7) that one could be called UTF-16LE, which is the native encoding in Windows since NT.

[milo1012]

BTW: The name of the Lister-option "Unicode" (#6) is IMHO not the best.
Also the help text ("Unicode ist ein Dateiformat, bei welchem jedes Zeichen 2 Bytes benötigt.") is a little bit imprecise. Unicode is a standard for encoding, representation, and handling of text, which an be implemented in different encodings, e.g. UTF-8, UTF-16, UTF-32 and others, like the UCS forms. Please also note, that the are also code points (characters), which need two 16-bit code units!

Indeed, this is bothering me since years.
+1

And while we're at it:

On the one side it's okay, if the font for ASCII only supports the characters of the 7 bit ASCII Code and respectively the font for ANSI supports all 256 characters of the 8 bit Code. However, the used code page is another point which has to be noticed.

Actually there is no such thing as an "ASCII" code page or charset. ASCII always refers to the well-known 7-bit characters which all code pages use as a least common denominator (besides things like EBCDIC, but we don't have this on a PC). In DOS ages, this was the so-called "OEM" code page (in IBM terms) to fill the 8th bit, in Windows this is the ANSI code page.
So I really wish that TC would stop speaking of "ASCII charset (DOS)", as this is just nonsense, but rather of sth like DOS/IBM/OEM charset, since even Microsoft refers to it as "OEM".

[browny]

Actually there is no such thing as an "ASCII" code page or charset. ASCII always refers to the well-known 7-bit characters

There were times when no Windows existed, before "ANSI". That is when such things as "extended ASCII" or "DOS code pages" did exist.
But of course TC could update to the current terminology.

In DOS ages, this was the so-called "OEM" code page (in IBM terms) to fill the 8th bit, in Windows this is the ANSI code page. So I really wish that TC would stop speaking of "ASCII charset (DOS)", as this is just nonsense, but rather of sth like DOS/IBM/OEM charset, since even Microsoft refers to it as "OEM".

In the current online MS Docs at least these code pages exist right now: ANSI, ANSI/OEM, EBCDIC, ISO, OEM PC and Open Systems.

Additionally,

and respectively the font for ANSI supports all 256 characters of the 8 bit Code.

This is an oversimplification to think that ANSI is 8-bit only. ANSI code pages for Oriental languages may have somewhat more that 256 characters.

[milo1012]

There were times when no Windows existed, before "ANSI". That is when such things as "extended ASCII" or "DOS code pages" did exist.

Of course, but Windows and the current API borrowed a huge amount of compatibility layers for the old (MS)DOS, that's why you'll see both pages together when using localization functions. But even when this was called "extended ASCII" at some point, this is not the same as calling it an ASCII charset.
And to quote
https://en.wikipedia.org/wiki/Extended_ASCII

The use of the term is sometimes criticized,[1][2][3] because it can be mistakenly interpreted to mean that the ASCII standard has been updated to include more than 128 characters or that the term unambiguously identifies a single encoding, neither of which is the case.

It's called OEM page because it's characters for the 8th bit were burned in the graphics ROM, and this set depended on the manufacturer (OEM) and/or region for which the IBM PC clone was sold.

In the current online MS Docs at least these code pages exist right now: ANSI, ANSI/OEM, EBCDIC, ISO, OEM PC and Open Systems.

These are just the theoretically available code pages for some conversion functions, some are not available at all, depending on the OS installation, but the whole MSDN refers to it as OEM, see e.g.
GetOEMCP
GetCPInfo
MultiByteToWideChar
WideCharToMultiByte
...

[rsa]

This is an oversimplification to think that ANSI is 8-bit only. ANSI code pages for Oriental languages may have somewhat more that 256 characters.

Never heard that. Please tell me/us more!
In my and the common understanding it's an 8 bit Code, like it's written in Wikipedia or , https://de.wikipedia.org/wiki/ANSI-Zeichencode

[ghisler(Author)]

Btw, I have tried Consolas, and it looks horrible. I don't think I can use that as the default font, it would create a sh*tstorm.

[RoMa]

Consolas seems to be designed with 'Clear Type' in mind. Without this type of font smoothing being active it's really a mess. On the other hand: I never operated an LCD display without 'Clear Type' since that was available.

[Dalai]

2ghisler(Author)
For PowerShell, but it's about Consolas font:

Another prerequisite is to enable ClearType on your LCD. If ClearType is not enabled then Consolas will actually look worse than using the default fonts!

Maybe you can detect whether or not ClearType is enabled and use Consolas only if that's the case? I don't know if there's a function for this; I can imagine that MS probably says "Don't try to detect it because your application shouldn't care about it, it's the user who sets what he wants" or some other nonsense...

Regards
Dalai

[rsa]

Oh, I don't know how Consolas looks without ClearType as I have ClearType activated on my Windows 10 (fairly new installation, no display options changed but colors; most settings still on default values) with a resolution of 1980 x 1080 pixels on a 23 " screen.

I just want you to show, how everything looks on my place.

First I've created a sample file with Windows Notepad (as UTF-8).
The 'Lorem ipsum' sentence was copied (from Wikipedia).
Then I've entered the following characters until the line with the umlauts - except the majuscule eszett, which I've copied from Wikipedia.
Then I've copied common special characters from Wikipedia (dashes, quotation marks).
At the end I've copied characters tables for ASCII, Latin-9 and Windows-1252 from Wikipedia.

Then I've opened the file with Lister (default settings).
ANSI
ASCII
Variable width

Then I've changed the font (just for variable width) to
Consolas and then to
Segoe UI.
Both in size 10 (which results in size 13) and standard format.

We all know: Beauty is in the eye of the beholder.
For me Consolas (and also Segoe UI) look good and for me it would be a useful replace of the default fonts ("Fixedsys" and "Terminal").


As @ghisler has an objection on the font Consolas - whether justified or not, I even more prefer a new default font (with good Unicode support for western characters, like Consolas) just for text files in Unicode (UTF-8, UTF-16, etc.; also for HTML files encoded in UTF-8) and let the actual default font (ANSI) for the other display options (Text, Binary, and Hex).

As I'd like to have a font with good support of western Unicode characters also in variable width, I'd also prefer a new font therefore, e.g. Segoe UI. Even this is not a problem to change the font for me in the options menu.

[browny]

Consolas seems to be designed with 'Clear Type' in mind. Without this type of font smoothing being active it's really a mess. On the other hand: I never operated an LCD display without 'Clear Type' since that was available.

A curious fact about different display types.
MS DOS users should remember standard 16 text colours. Recently MS claimed that is was optimized for CRT displays, and slightly changed the colours to look better on LCD screens.