[OT] How to check if a file is ASCII or UNICODE

[tbeu]

I need to check if a file is plain ASCII or UNICODE encoded. There is the WIN32 API function IsTextUnicode which I believe is only working if _UNICODE is defined. But how can I check if _UNICODE is not defined?

Thanks,
tbeu

[Alextp]

2tbeu
I think it should work in all cases. Where did you see requirement of "_UNICODE"?

[Lefteous]

2tbeu

I need to check if a file is plain ASCII or UNICODE encoded.

In this case I would just check the BOM:
http://unicode.org/unicode/faq/utf_bom.html#22

[Alextp]

2Lefteous
BOM may be absent in text. See desc of IsUnicodeText: it performs ont only BOM check but also some adv tests.

[tbeu]

Where did you see requirement of "_UNICODE"?

I got it from here. However, it is said that unicode must be defined, not _UNICODE.

[Alextp]

It must be defined for MFC app to be Unicode app. The func IsTextUnicode doesn't depend on this; just give it a stream of bytes and it works.

[tbeu]

Alex, maybe you can again help me. In fact, I want to rebuild TC's check of text and binary files. ASAIK, TC checks the first 8kiB of a file for characters 0x00 to 0x05. But this way, UNICODE files are detected as text whereas I need to consider them as binary data.

[Alextp]

But this way, UNICODE files are detected as text whereas I need to consider them as binary data.

You mean they are detected as binary but you need them as text?

You can call IsTextUnicode to determine Unicode. When result is false you can use check for bytes 0x00 to 0x05...
Is this not enought.

[tbeu]

You mean they are detected as binary but you need them as text?

No, the other way round. I need them as binary data, not as text. Checking for 0x00 to 0x05 always fails, when I open/read the file by fopen/fread/fgetc or CreateFile/ReadFile. Such, UNICODE files are always detected as text which is not what I want.

[Alextp]

Checking for 0x00 to 0x05 always fails, when I open/read the file by fopen/fread/fgetc or CreateFile/ReadFile.

It must not; I also use CreateFile/ReadFile and check is ok. You can see source of my Viewer component: functions IsFileUnicode/IsFileText perform this check ok.