There are MAJOR changes in select list handling in TC

[louwin]

For any of you worrying about funny bugs etc in your scripts etc....

A major Windows 11 update happened last night (well in Australia anyway)! TC used to output select lists in utf-16-le (yesterday) - today TC outputs ASCII select lists.

Just a warning....

[white]

What do you mean with select lists?

[petermad]

I Guess louwin means lists generated with the SAVESELECTION command

There is possibility of forcing UTF-16.with SAVESELECTIONW

17: SAVESELECTION, SAVESELECTIONA, SAVESELECTIONW: save names of selected files to the specified file. With character A at the end, the list is saved as ANSI, with W as Unicode UTF-16. Without suffix, it depends on the operating system (NT-based systems use Unicode).

18. SAVEDETAILS, SAVEDETAILSA, SAVEDETAILSW: save names and details of selected files to the specified file. With character A at the end, the list is saved as ANSI, with W as Unicode UTF-16. Without suffix, it depends on the operating system (NT-based systems use Unicode). Append H after DETAILS to save with headers.

I guess it is the part in bold here above that is affected.

[louwin]

You know - right click on a/some file(s) and then pass it to a TC button using %L. TC then writes a CMD999.tmp file and passes it to the function/(script/App/exe/batch/Python/AHK etc etc. I have a complex script (well MANY actually) that rely on a utf-16le select list format. If your process gets an ASCII select list when it is expecting a utf-16-le it displays a weird construct Mojibake (Garbage Chinese characters).

Code like "is_file = os.path.isfile(full_path)" fail and may even KILL the script silently with NO log, NO explanation, NO nothing etc.

[Horst.Epp]

No such bug here.
The actual Windows update didn't change anything in TC behaviour.
Using %WL still writes files with UTF-16 without problems.
Using %L writes ANSI files as designed.

[petermad]

2louwin

I don't think that is a new Windows 11 thing - My windows 7 and 10 also write the CMD*.tmp tile in ANSI when using %L - you need to use %WL to get UTF-16

[Dalai]

%L has always been writing ANSI text. Use either %WL for UTF-16 or %UL for UTF-8. Both are with BOM, so you might need to keep the receiving program's capabilities in mind. IIRC many CMD tools can't process such files properly (the type command can).

[louwin]

%L has always been writing ANSI text. Use either %WL for UTF-16 or %UL for UTF-8. Both are with BOM, so you might need to keep the receiving program's capabilities in mind. IIRC many CMD tools can't process such files properly (the type command can).

Sorry! But I have been using %L AND getting utf-16-le for MONTHS! And did on 1st December. On the 2nd December (after the Windows 11 24H2/25H2? MAJOR update) ALL my select lists swapped to ANSI! I'm using petermad's Extended TC so it maybe something in THAT setup????
BUT You ARE right though! I changed to %UL and got utf-8-sig. I changed to %WL and got back utf-16-le! I'll NOW change my default to %WL. THANKS FOR THAT Dalai

So, BASICALLY, THE Windows 11 update must have swapped some default setting!?

Edit: By changing to %WL fixed ALMOST everything! Except it now passes the xFF xFE (BOM) as part of the first file's path!!!! According to the AI you then have to change your encoding to utf-16 NOT utf-16-le (in your script)!

[petermad]

I'm using petermad's Extended TC so it maybe something in THAT setup?

The only changes that using my Extenced Menus causes to TC's .ini file are:these two parameters in the [Configuration] section:
LanguageIni=
Mainmenu=

In the menu I only use %L in one item: "Files" -> "Copy (Names)" -> "Copy Path + File Names to the File SELECTEDFILES.txt (Force ANSI) *"

There is absolutely no way that either of these can influence how TC handles %L.

[louwin]

There is absolutely no way that either of these can influence how TC handles %L.

Thanks for the confirmation petermad.

So ALL I can say is - on 1st December all my scripts using %L worked

on 2nd December ALL scripts using %L didn't work! They just die silently.

Any Python scripts with "try/except with open" fail error 123. Examination of the paths show "㩸坜牯䉫湥档䑜䍄牵敲瑮楆敬吠敨爠獯⁥㼿攮"

Judicious use of %WL and utf-16 (NOT utf-16-le) has restored order to my scripts....

[petermad]

2louwin
Could you maybe have been using Unicode as your Windows System codepage, and then the update at Dec. 2. has changed that to the codepage of your locale? I don't know if that would influence the content of ANSI files. And it looks like the Wndows beta unicode codpage setting only sets codepage 65001 (UTF-8)

https://sema4.ai/docs/automation/troubleshooting/windows-codepage

[louwin]

2louwin
Could you maybe have been using Unicode as your Windows System codepage, and then the update at Dec. 2. has changed that to the codepage of your locale? I don't know if that would influence the content of ANSI files. And it looks like the Wndows beta unicode codpage setting only sets codepage 65001 (UTF-8)

Again, sorry. I'm a retired mainframe analyst/programmer, mostly Unix. I read the article you pointed at and didn't get much from it as I'm Unix based!

I passed the article's comment to my AI and his? response....

Ledger closure
“Petermad’s suggestion flips Windows’ default code page from cp1252 to UTF‑8. Ritual truth: legacy apps see UTF‑8 instead of ANSI, reducing hungry glyphs but risking breakage.”
So in plain terms: it’s a system‑wide switch that tells Windows “stop pretending the world is cp1252, treat it as UTF‑8 instead.” That can help with your select list anomalies, but it’s a blunt instrument — it changes behavior for every non‑Unicode app on the machine.

My problem is - my script tries to os.rename filenames with what we call "hungry glyphs". These glyphs do funny things! For example, if a glyph is in front of "ﲫ.mp4" it "eats" the full stop!

Sorry! I'll stop before I get too specific and too boring for most....

By the way! Thanks for your extended TC

[Dalai]

Edit: By changing to %WL fixed ALMOST everything! Except it now passes the xFF xFE (BOM) as part of the first file's path!!!! According to the AI you then have to change your encoding to utf-16 NOT utf-16-le (in your script)!

Another case of where the output of an "AI" can't be accepted at face value. FF FE is the BOM for UTF-16 LE (Little Endian). If it were FE FF it would be Big Endian. As far as I know everything on Windows is UTF-16 LE (networking and maybe some other stuff excluded), meaning that if the endianness isn't specified or the BOM is missing, LE is assumed. See Wikipedia for example:

If the BOM is missing, RFC 2781 recommends[e] that big-endian (BE) encoding be assumed. In practice, due to Windows using little-endian (LE) order by default, many applications assume little-endian encoding.

In the end it depends on the application that processes the list generated by TC. Some programs just can't deal with BOMs, as I eluded to earlier.

[louwin]

As far as I know everything on Windows is UTF-16 LE

Yes! Possibly the case TILL 01:00 morning of the 2nd December, in Perth, Western Australia anyway! That's when 24H2/25H2? was loaded. Early morning I switched on and didn't realize it BUT the default had changed to ANSI! As I said 01/12 %L was utf-16-le - early morning 02/12 %L was ANSI!!!! One of my scripts had NOT been affected AT ALL as it ONLY used ANSI not Unicode. Other scripts fell in a heap!!!!

[white]

As I said 01/12 %L was utf-16-le - early morning 02/12 %L was ANSI!!!! One of my scripts had NOT been affected AT ALL as it ONLY used ANSI not Unicode. Other scripts fell in a heap!!!!

If, as you say, %L produced utf-16-le, then you would have had the same problems as when using %WL now. The BOM would have been a problem back then as well and your script expecting ANSI probably wouldn't have worked. It seems as though petermad is right and you made the wrong assumption and the problem has to do with the codepage being used.