[TC9.51] LOADSEARCH/LOADSELECTION does not work over utf-8 file list w/o BOM

[andry81]

OS: Windows x64

Steps to repro:

1. create utf-8 file list w/o BOM.
2. create a file with a trademark character (U+2122) in the file name (use charmap application to lookup character and copy it to the clipboard).
3. add the file to the file list.

If add the BOM it starts to work, which means I have to always prefix the BOM into the list.

Can you remove that requirement and always treat the list file as utf-8 file even if BOM is not prefixed?

[ghisler(Author)]

Please use UTF-16 instead, it always has BOM.

The problem is that some characters valid in UTF-8 are also valid in Chinese or Korean ANSI.

[andry81]

I am not quite get it. Why not to force treat it as utf-8?

[Fla$her]

UTF-16 ... always has BOM.

It's doesn't correspond to reality.

[Usher]

Why not to force treat it as utf-8?

Because of backward compatibility.

However, there might be an option in wincmd.ini for 8-bit text files without BOM - something like "Define charset by file type" which would allow to choose at least Autodetect/ANSI/DOS/UTF-8 for specific file extensions.

Notice that both Unicode and its UTF-8 form aren't perfect in design. That's why automatic detection fails in some cases and we must live with it.

[andry81]

Why not to force treat it as utf-8?

Because of backward compatibility.

Backward compatibility with DOS? The file system in the Windows already Unicode.

However, there might be an option in wincmd.ini for 8-bit text files without BOM - something like "Define charset by file type" which would allow to choose at least Autodetect/ANSI/DOS/UTF-8 for specific file extensions.

This has no sense, a windows files path already in unicode.

Notice that both Unicode and its UTF-8 form aren't perfect in design. That's why automatic detection fails in some cases and we must live with it.

Automatic detection of that? Why you need automatic detection of unicode characters?

[Usher]

Why TC should stop work in Windows 95? The file system is in UTF-16, not in UTF-8.

And a file list is just a text file. How TC would know that you'd like to deal with a very specific kind of text?

[andry81]

Why TC should stop work in Windows 95? The file system is in UTF-16, not in UTF-8.

How is it related to the question about the BOM presence?

And a file list is just a text file. How TC would know that you'd like to deal with a very specific kind of text?

To not know it the unicode is exist in first place.

[Usher]

I'm afraid I don't understand your English. I can't help you.

[andry81]

I'm afraid I don't understand your English. I can't help you.

let me help you: translate.google.com

[Hacker]

Moderator message from: Hacker

andry81,
Please stay polite.