Why does TC create lists of files (%L %UL) always with BOM?

[PiotrMPPL]

Hi

Why does TC create lists of files (%L %UL) always with BOM?

[Horst.Epp]

The detection of character sets in different formats is easier with BOM.
I never had problems with this, in a lot of file managers, editors and other tools.
What's your specific problem with it ?

[Dalai]

%L shouldn't add a BOM because it saves the list file as ANSI (without a BOM as far as I know). I suppose you mean %UL and %WL.

If a BOM exists, there is no doubt on how to interpret the following data. If there is no BOM certain characters can be ambiguous because they're valid in UTF-8 and ANSI simultaneously for example. BTW, it's documented in TC's help that %UL and %WL save the list with a BOM.

Regards
Dalai

[PiotrMPPL]

How to correctly read %UL list from BOM in windows batch?

I need to read the first line from the %UL list without BOM.

[ghisler(Author)]

Just cut off the first character, e.g. something like %var:~1%

[nsp]

Windows batch in console do not handle Unicode correctly. You can anyhow handle some input/output in UTF-8 changing codpage to 65001.
To print, you must change console font to lucida console or one with all the char you plan to print.

UTF-16 is not supported at all even with codepage !

If you want to use utf-16, do not use batch. You can use powershell with Unicode encoding.

[milo1012]

This was actually discussed in the past several times.
I even made a suggestion to include BOM-less UTF-8 list files, years ago
viewtopic.php?t=48032
Not implemented so far, from what I can see.

Windows batch in console do not handle Unicode correctly. You can anyhow handle some input/output in UTF-8 changing codpage to 65001.
To print, you must change console font to lucida console or one with all the char you plan to print.

UTF-16 is not supported at all even with codepage !

If you want to use utf-16, do not use batch. You can use powershell with Unicode encoding.

Just a small reminder :
You can type a UTF-16 list file (for loops) w/o problems in batch/cmd:
viewtopic.php?t=48032&p=335753

[nsp]

Just a small reminder :
You can type a UTF-16 list file (for loops) w/o problems in batch/cmd:
viewtopic.php?t=48032&p=335753

On this thread, the only working cases are using UTF-8.

Just give a try with cmd /U with an utf-16LE file i.e test.lst
cmd /U only produce UTF-16LE output !
do a loop like

Code: Select all

for /F "usebackq tokens=* delims=" %n in ( test.lst ) do notepad "%n"

See if notepad open each unicode file name !