Lister: default font (for Unicode/UTF-8)

Here you can propose new features, make suggestions etc.

Moderators: white, Hacker, petermad, Stefan2

Post Reply
rsa
Junior Member
Junior Member
Posts: 57
Joined: 2003-11-19, 20:41 UTC

Lister: default font (for Unicode/UTF-8)

Post by *rsa »

The integrated Lister uses old fonts as default fonts: "Fixedsys" for ANSI and "Terminal" for ASCII.
The default font for variable width is "System" (bold). This font only exists as bold and bold-italic.

First I'd like to say, that all these three fonts are quite old and IMHO not very nice. The font "System" is not available in a standard font (not bold, not italic).
These old fonts also have shortcomings on common Unicode characters, e. g. the en dash ("Halbgeviertstrich", Unicode code point U+2013, UTF-8 hex e2 80 93).

On the one side it's okay, if the font for ASCII only supports the characters of the 7 bit ASCII Code and respectively the font for ANSI supports all 256 characters of the 8 bit Code. However, the used code page is another point which has to be noticed.

On the other side there is the behavior on Unicode-files. If a file is encoded in UTF-8 (and has a Byte Order Mark for UTF-8), Lister (with default "Autodetect") automatically activates the display option "UTF-8" and ANSI. So the font Fixedsys is used, but this font hat no full support of Unicode; it also does not support widely used characters, like the en dash.
If the default font would be a font with better Unicode support, like Consolas, also the en dash will be displayed correct.

Another proper way might be, to change the Lister and define also a font for Unicode (all UTF-formats), where Consolas could be a good option as default font. The shortcut (key) "U" might be a good way for this. So it's possible to have a proper Unicode font for monospace (default) and a way to switch to a proportional font with key "V".
The good default font for variable width might be "Segoe UI".

Consolas is the default monospace font in Windows, since Windows Vista, i.e. since 2007!
I think, it's time to use this font also for the Lister as default font.


Of course, everyone can configure his own fonts, but there seems to be a problem (?) with the font size.
If I chose Consolas, Standard, Size 11 in the config dialog "Schriftart", it will be displayed as "Consolas,-16" in config dialog "Configure Lister". It's the same of all fonts and all sizes: the used sized is bigger than the selected size. For me this is a little bit strange.



BTW: The name of the Lister-option "Unicode" (#6) is IMHO not the best.
Also the help text ("Unicode ist ein Dateiformat, bei welchem jedes Zeichen 2 Bytes benötigt.") is a little bit imprecise. Unicode is a standard for encoding, representation, and handling of text, which an be implemented in different encodings, e.g. UTF-8, UTF-16, UTF-32 and others, like the UCS forms. Please also note, that the are also code points (characters), which need two 16-bit code units!


P.S.: Windows 10 Pro (64 Bit, DE) with Total Commander 9.12 (64 Bit).
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

Sounds like a good idea to use Consolas for fixed pitch fonts, if it is installed. I will check whether I can add it.
Author of Total Commander
https://www.ghisler.com
browny
Senior Member
Senior Member
Posts: 287
Joined: 2007-09-10, 13:19 UTC

Re: Lister: default font (for Unicode/UTF-8)

Post by *browny »

rsa wrote: BTW: The name of the Lister-option "Unicode" (#6) is IMHO not the best.
Following the logic of the next menu item UTF-8 (#7) that one could be called UTF-16LE, which is the native encoding in Windows since NT.
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Re: Lister: default font (for Unicode/UTF-8)

Post by *milo1012 »

rsa wrote:BTW: The name of the Lister-option "Unicode" (#6) is IMHO not the best.
Also the help text ("Unicode ist ein Dateiformat, bei welchem jedes Zeichen 2 Bytes benötigt.") is a little bit imprecise. Unicode is a standard for encoding, representation, and handling of text, which an be implemented in different encodings, e.g. UTF-8, UTF-16, UTF-32 and others, like the UCS forms. Please also note, that the are also code points (characters), which need two 16-bit code units!
Indeed, this is bothering me since years.
+1

And while we're at it:
rsa wrote:On the one side it's okay, if the font for ASCII only supports the characters of the 7 bit ASCII Code and respectively the font for ANSI supports all 256 characters of the 8 bit Code. However, the used code page is another point which has to be noticed.
Actually there is no such thing as an "ASCII" code page or charset. ASCII always refers to the well-known 7-bit characters which all code pages use as a least common denominator (besides things like EBCDIC, but we don't have this on a PC). In DOS ages, this was the so-called "OEM" code page (in IBM terms) to fill the 8th bit, in Windows this is the ANSI code page.
So I really wish that TC would stop speaking of "ASCII charset (DOS)", as this is just nonsense, but rather of sth like DOS/IBM/OEM charset, since even Microsoft refers to it as "OEM".
TC plugins: PCREsearch and RegXtract
browny
Senior Member
Senior Member
Posts: 287
Joined: 2007-09-10, 13:19 UTC

Re: Lister: default font (for Unicode/UTF-8)

Post by *browny »

milo1012 wrote:Actually there is no such thing as an "ASCII" code page or charset. ASCII always refers to the well-known 7-bit characters
There were times when no Windows existed, before "ANSI". That is when such things as "extended ASCII" or "DOS code pages" did exist.
But of course TC could update to the current terminology.
milo1012 wrote:In DOS ages, this was the so-called "OEM" code page (in IBM terms) to fill the 8th bit, in Windows this is the ANSI code page. So I really wish that TC would stop speaking of "ASCII charset (DOS)", as this is just nonsense, but rather of sth like DOS/IBM/OEM charset, since even Microsoft refers to it as "OEM".
In the current online MS Docs at least these code pages exist right now: ANSI, ANSI/OEM, EBCDIC, ISO, OEM PC and Open Systems.

Additionally,
rsa wrote:and respectively the font for ANSI supports all 256 characters of the 8 bit Code.
This is an oversimplification to think that ANSI is 8-bit only. ANSI code pages for Oriental languages may have somewhat more that 256 characters.
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Re: Lister: default font (for Unicode/UTF-8)

Post by *milo1012 »

browny wrote:There were times when no Windows existed, before "ANSI". That is when such things as "extended ASCII" or "DOS code pages" did exist.
Of course, but Windows and the current API borrowed a huge amount of compatibility layers for the old (MS)DOS, that's why you'll see both pages together when using localization functions. But even when this was called "extended ASCII" at some point, this is not the same as calling it an ASCII charset.
And to quote
https://en.wikipedia.org/wiki/Extended_ASCII
The use of the term is sometimes criticized,[1][2][3] because it can be mistakenly interpreted to mean that the ASCII standard has been updated to include more than 128 characters or that the term unambiguously identifies a single encoding, neither of which is the case.
It's called OEM page because it's characters for the 8th bit were burned in the graphics ROM, and this set depended on the manufacturer (OEM) and/or region for which the IBM PC clone was sold.
browny wrote:In the current online MS Docs at least these code pages exist right now: ANSI, ANSI/OEM, EBCDIC, ISO, OEM PC and Open Systems.
These are just the theoretically available code pages for some conversion functions, some are not available at all, depending on the OS installation, but the whole MSDN refers to it as OEM, see e.g.
GetOEMCP
GetCPInfo
MultiByteToWideChar
WideCharToMultiByte

...
TC plugins: PCREsearch and RegXtract
rsa
Junior Member
Junior Member
Posts: 57
Joined: 2003-11-19, 20:41 UTC

Re: Lister: default font (for Unicode/UTF-8)

Post by *rsa »

browny wrote:This is an oversimplification to think that ANSI is 8-bit only. ANSI code pages for Oriental languages may have somewhat more that 256 characters.
Never heard that. Please tell me/us more!
In my and the common understanding it's an 8 bit Code, like it's written in Wikipedia or , https://de.wikipedia.org/wiki/ANSI-Zeichencode
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

Btw, I have tried Consolas, and it looks horrible. I don't think I can use that as the default font, it would create a sh*tstorm.
Author of Total Commander
https://www.ghisler.com
User avatar
RoMa
Junior Member
Junior Member
Posts: 25
Joined: 2003-03-08, 11:38 UTC

Post by *RoMa »

Consolas seems to be designed with 'Clear Type' in mind. Without this type of font smoothing being active it's really a mess. On the other hand: I never operated an LCD display without 'Clear Type' since that was available.
User avatar
Dalai
Power Member
Power Member
Posts: 9364
Joined: 2005-01-28, 22:17 UTC
Location: Meiningen (Südthüringen)

Post by *Dalai »

2ghisler(Author)
For PowerShell, but it's about Consolas font:
Another prerequisite is to enable ClearType on your LCD. If ClearType is not enabled then Consolas will actually look worse than using the default fonts!
Maybe you can detect whether or not ClearType is enabled and use Consolas only if that's the case? I don't know if there's a function for this; I can imagine that MS probably says "Don't try to detect it because your application shouldn't care about it, it's the user who sets what he wants" or some other nonsense...

Regards
Dalai
#101164 Personal licence
Ryzen 5 2600, 16 GiB RAM, ASUS Prime X370-A, Win7 x64

Plugins: Services2, Startups, CertificateInfo, SignatureInfo, LineBreakInfo - Download-Mirror
rsa
Junior Member
Junior Member
Posts: 57
Joined: 2003-11-19, 20:41 UTC

Post by *rsa »

Oh, I don't know how Consolas looks without ClearType as I have ClearType activated on my Windows 10 (fairly new installation, no display options changed but colors; most settings still on default values) with a resolution of 1980 x 1080 pixels on a 23 " screen.

I just want you to show, how everything looks on my place.

First I've created a sample file with Windows Notepad (as UTF-8).
The 'Lorem ipsum' sentence was copied (from Wikipedia).
Then I've entered the following characters until the line with the umlauts - except the majuscule eszett, which I've copied from Wikipedia.
Then I've copied common special characters from Wikipedia (dashes, quotation marks).
At the end I've copied characters tables for ASCII, Latin-9 and Windows-1252 from Wikipedia.

Then I've opened the file with Lister (default settings).
ANSI
ASCII
Variable width

Then I've changed the font (just for variable width) to
Consolas and then to
Segoe UI.
Both in size 10 (which results in size 13) and standard format.

We all know: Beauty is in the eye of the beholder.
For me Consolas (and also Segoe UI) look good and for me it would be a useful replace of the default fonts ("Fixedsys" and "Terminal").


As @ghisler has an objection on the font Consolas - whether justified or not, I even more prefer a new default font (with good Unicode support for western characters, like Consolas) just for text files in Unicode (UTF-8, UTF-16, etc.; also for HTML files encoded in UTF-8) and let the actual default font (ANSI) for the other display options (Text, Binary, and Hex).

As I'd like to have a font with good support of western Unicode characters also in variable width, I'd also prefer a new font therefore, e.g. Segoe UI. Even this is not a problem to change the font for me in the options menu.
browny
Senior Member
Senior Member
Posts: 287
Joined: 2007-09-10, 13:19 UTC

Post by *browny »

RoMa wrote:Consolas seems to be designed with 'Clear Type' in mind. Without this type of font smoothing being active it's really a mess. On the other hand: I never operated an LCD display without 'Clear Type' since that was available.
A curious fact about different display types.
MS DOS users should remember standard 16 text colours. Recently MS claimed that is was optimized for CRT displays, and slightly changed the colours to look better on LCD screens.
Post Reply