"Synchronize; Add current directories and settings to list" problem when List name has utf8 characters

[tm8544]

When Synchronize list name has utf-8 characters, that list name is shown 4 times. See image below.
Image: https://pasteboard.co/G0SKPnkZP3Or.png

TC 11.51 x64

[AntonyD]

It's strange, but I also have a problem with the repeatability of entries - it's the only one in the folder creation dialog (F7).
In the history drop-down list of this process, I have entries repeated exactly the same 4 times.
At first, I thought about creating a post about this, but then I couldn't easily reproduce this case on a clean configuration file.
Now, with this bug, you have raised the desire to find my conditions for reproducing of my bug)))

[white]

Not confirmed when using the name "Tämä ei toimi". Does it always happen? Does it happen if you test it with a clean configuration file?

[tm8544]

I made some tests to figure out why I see the problem and you don't.

It is not a case of TC configuration.
I have set WIndows 11 to use UTF8 character set, see image below.
If I uncheck that feature, there is no problem with TC.

Image: https://pasteboard.co/mFsuiCKGmLSM.png

The reason I have checked it is that this way I can use utf8 charset in .cmd files.
I sincerely hope that even though this is a Beta feature in Windows, you would look into this.
On some languages, like Finnish, it is quite impossible to manage without using utf8-characters.

[Horst.Epp]

The reason I have checked it is that this way I can use utf8 charset in .cmd files.
I sincerely hope that even though this is a Beta feature in Windows, you would look into this.
On some languages, like Finnish, it is quite impossible to manage without using utf8-characters.

The normal solution for this problem is to use the following command in your cmd
which changes the code page for this to UTF-8
chcp 65001

The way you do it causes problems in many Windows tools.

[tm8544]

I think that changing code page is not a solution but a way to divert the problem.
I have had this problem so far with one Windows tool only (TC).

[Horst.Epp]

I think that changing code page is not a solution but a way to divert the problem.

It's the normal way to work with cmd and Unicode and has nothing to do with diverting a problem.
No problems ever reported here or on other sites.
What's the problem to add one line in your cmd scripts ?

[white]

Hypothesis:

When the Windows Beta feature "Use UTF-8 for worldwide language support" is disabled, TC converts the input to the system's ANSI code page (e.g., 1252 for Finnish). If you enter a character not supported by that code page (like a smiley ), the conversion fails, and TC shows the error "Unfortunately this function doesn’t support Unicode characters!" This suggests the "Synchronize dirs" settings function relies on ANSI, not Unicode.

When the Beta feature is enabled, the system code page becomes 65001 (UTF-8), which supports all Unicode characters. The conversion no longer fails, so TC doesn’t show the error but receives multi-byte UTF-8 (e.g., "ä" as two bytes), causing the repetition bug.

To address this, a fix could be for TC to check the system code page. If it’s 65001, TC could convert the input to a fixed single-byte code page (e.g., 1252) instead of using UTF-8. This would keep the input single-byte, avoid the bug, and still reject unsupported characters consistently.

I'm just guessing here. I'm not sure if this is how it all works.

[Dalai]

Enabling that option system-wide changes how the Windows API handles calls to ANSI functions (the ones with the trailing A in their name). If the option is disabled it behaves like it was in the past, i.e. an ANSI string is an ANSI string. If the option is enabled, the supposed ANSI strings become UTF-8 strings. This applies to all applications which call these ANSI API functions. It has no impact on the Unicode API functions (the ones with the trailing W in their name).

You're in luck if you haven't had problems with other programs so far, but that doesn't mean it will stay that way. A lot of programs don't expect such strings to have characters outside of the ANSI charsets in it. This can be especially problematic for older programs or ones that won't be updated anymore.

And, Horst.Epp has a point, changing the codepage to 65001 in a batch script is the usual way to go about UTF-8 characters in input/output files.

[ghisler(Author)]

When using UTF-8 as the system codepage, you will get into troubles when using plain text ini files, because Windows doesn't support UTF-8 in ini files.

Try opening your wincmd.ini with notepad (F4) and save it as UTF-16LE.

[ghisler(Author)]

Unfortunately my advice doesn't help here because the problem is caused by a bug in Windows itself!

I'm using the (ANSI) Windows function GetMenuString to get the titles of existing menu items to check whether they already exist or not.
This function returns less characters than it should. In your example, instead of
"Tämä ei toimi"
it returns two characters less:
"Tämä ei toi"
The reason seems to be that it counts the number of characters (ä counts as one) and not the number of bytes (ä is represented by 2 bytes in UTF-8).
Therefore it returns 2 characters less than it should.

Fortunately I can call the Unicode version GetMenuStringW and get the correct value back, so I can fix this.