Solved [TC8b2 x64] UFT-8 files are not recognized by Lister

[petermad]

UTF-8 files with BOM marker (for example TC's log files) are not automatically recognized as such by x64 lister.

Actually TC's auto-detection of UTF-8 files without BOM doesn't work either.

[karlchen]

Hello, petermad.

UTF-8 files with BOM marker (for example TC's log files) are not automatically recognized as such by x64 lister.

Cannot confirm this statement.
Created a UTF-8 file with BOM using Notepad2.
T.C. 8.0ß2 32-bit lister identified the file correctly as UTF-8.
So did T.C. 8.0ß2 64-bit lister, i.e. it identified the file correctly as UTF-8.

Karl
--
Server 2008 R2
Total Commander 8.0ß2 64-bit
Total Commander 8.0ß2 32-bit

[karlchen]

Hello, petermad.

Actually TC's auto-detection of UTF-8 files without BOM doesn't work either.

Well, cannot confirm this, either.
Created a UTF-8 file using Notepad2.

Both listers, T.C. 8.0.ß2 32-bit and 64-bit identified it correctly as UTF-8.

Perhaps the point is that my files held non English characters which "helped" lister to realize the files were UTF-8.

Karl

[karlchen]

Ok, created a UTF-8 textfile using Notepad2.
The file holds only English characters.
The file does not have a BOM.

Both listers, T.C. 8.0ß2 32-bit and 64-bit, fail to recognize the file as being UTF-8.

--

Converted the file mentioned above into a UTF-8 file with BOM.
The file holds only English characters.
The file does have a BOM.

Both listers, T.C. 8.0ß2 32-bit and 64-bit, correctly recognize the file as being UTF-8.

--

Conclusion:

Lister needs some kind of hint that a file is UTF-8, either the BOM, or it must hold some non English characters.

Cheers,
Karl

[petermad]

I have uploaded a TC logfile here: http://madsenworld.dk/tcmd/totalcmd.log

In my TC 8.0b2 x64 Lister it looks like this when loading: http://madsenworld.dk/tcmd/utf8-fail.png

I have to manually select UTF-8 (press 7) to have it shown correctly.

The same applies to http://madsenworld.dk/tcmd/totalcmd-nobom.log

With TC 8.0b2 x86 Lister recognizes the two files as UTF-8 automatically.

Notice that there is one non-English multibyte-character in the two files in line 2 pos 33 (an ø) - that should make it possible to detect the file as UFT-8 even without BOM.

[karlchen]

Hello, petermad.

Downloaded your two sample files.

Confirmed:
The lister of Total Commander 8.0ß2 32-bit recognizes both files correctly as UTF-8 automatically.

Still Not Confirmed:
The lister of Total Commander 8.0ß2 64-bit recognizes both files correctly as UTF-8 automatically as well.

Tried it even with a fresh wincmd.ini file (totalcmd.64.exe /N /I=.\fresh.ini). Same result. The internal lister identified both files as UTF-8 automatically.

Crucial Setting:

Lister is configured to "autodetect" "font at startup".
If I set this to "Ansi", I get the reported behaviour.
(Inside lister go to options => configure => font at startup and set it to automatic. Problem solved.)

Server 2008 R2
Windows 2003 Std SP2 64-bit
Total Commander 8.0ß2 64-bit
(Even beta1 64-bit succeeded as well)

Cheers,
Karl

[ghisler(Author)]

Ok, created a UTF-8 textfile using Notepad2.
The file holds only English characters.
The file does not have a BOM.

That's a good joke!
An UTF-8 file without BOM and without any non-English characters is 100% identical to an ANSI file with the same contents.

2petermad
You problably have a different problem reported earlier: Lister's configuration dialog fails to show the startup file type option correctly. Try checking the file type (uppermost option) in the 32-bit version.

[petermad]

2ghisler(Author)

I guess you are right - x64 TC automatically sets The Lister option for "Display at startup" to "Text" when the configuration dialog is opened - even though the setting is actually "Autodetect" - and so if I doesn't notice that, when I go in to change something else in the configuration dialog, then "Text" mode is also saved without knowing it.

Workaround: Set "Display at startup" to "Autodetect" every time Listers setup dialog is used.

[ghisler(Author)]

This should be fixed now in beta 3, please test!

[karlchen]

Hello, Christian.

Provided that this is the source of the problem:

I guess you are right - x64 TC automatically sets The Lister option for "Display at startup" to "Text" when the configuration dialog is opened

then I can reproduce the problem using the internal lister of Total Commander 8.0 beta2 64-bit.

And I can cofirm that the internal lister of Total Commander 8.0 beta3 64-bit does not exhibit the same (mis)behaviour.

Cheers,
Karl
--
Total Commander 8.0 beta3 64-bit
Server 2008 R2

[ghisler(Author)]

Yes, that was the reason - thanks for confirming the fix!

[petermad]

Confirmed fixed in TC 8.0b3