Encoding of custom column language files and pluginst.inf - Unicode?

[Dalai]

Ah, now it makes sense.

Regards
Dalai

[Usher]

2ghisler(Author)
I'm afraid it's a total mess with Unicode support:

• LNG file may be in ANSI or UTF-8 with BOM, UTF-16 is not supported;
• MNU files may be in ANSI or UTF-8 with NO BOM only, UTF-16 is not supported;
• INI files may be in ANSI or UTF-16, UTF-8 support is broken…
• What about INF, INC, BAR and other files?

Why can't we have a common solution for all text config files?

[ghisler(Author)]

mnu and plugin ini files need to use the same encoding as the main lng file.

[MVV]

It would still be much better if TC could respect pure-Unicode INI files (which are 100% valid in Windows) and read them using pure-Unicode API...

[petermad]

2ghisler(Author)
Does that apply to pluginst.inf too ?

[ghisler(Author)]

It would still be much better if TC could respect pure-Unicode INI files (which are 100% valid in Windows) and read them using pure-Unicode API...

While I could add this as a feature, the problem is that thousands of people will continue to use older versions for many years. Such language files wouldn't work for those users.

[MVV]

Well, wincmd.ini encoding may be safely changed, but for plugin LNG files you're right, this will require newest TC, so perhaps you could use ulng extension for UTF-16 files, or just leave this on users (update TC or edit LNG files). But this is the feature, so we need it...

Following parameters may be used for indication:
+ codepage=1200 for TC LNG files (UTF-16 codepage number)
+ codepage=1200 for wincmd.ini in [Configuration] section
+ some kind of similar parameter for plugin LNG files (e.g. codepage=1200 before all sections for manual parsing or in some section like [encoding] or [lng] for reading via API)

So all old LNG files would work as expected but new ones could use UTF-16.

[Usher]

2MVV
Codepage declaration is needed mostly for 8-bit charsets, including UTF-8, as BOM is not required in this case.
You don't need codepage declaration in UTF-16 files. They all must use BOM. The problem is that TC in some situations doesn't recognize BOM and ignores files with embedded 0x00 bytes. They are unreadable so codepage declaration inside such files is also ignored.

IMHO it's highest time to start working on fully Unicode Total Commander X, where X stands for number 10 and XP+ compatibility.

[MVV]

Usher,
Direct UTF-16 indication is needed for TC to know that this INI (or LNG) must be read using Unicode API and not ANSI API. Windows allows transparently reading both UTF-16 and ANSI INI files with both Unicode and ANSI API (with automatic conversation according to system codepage if needed), and TC currently reads all INI files as ANSI with manual codepage conversation, but in case of UTF-16 INI manual conversation doesn't allow mixing languages/codepages in the same UTF-16 INI file.

[Dalai]

Well, in case of the suggestions to use a second file (*W.inf and *W.lng, or *.ulng) it's already indicated to TC that it should use Unicode API to read it. Don't you think such indicator is sufficient?

Regards
Dalai

[MVV]

Yes, suffixed name (or prefixed/suffixed extension) is sufficient to indicate that file must be read as pure Unicode, of course.