LOADLIST command and UTF8 file lists

English support forum

Moderators: white, Hacker, petermad, Stefan2

User avatar
Horst.Epp
Power Member
Power Member
Posts: 6450
Joined: 2003-02-06, 17:36 UTC
Location: Germany

LOADLIST command and UTF8 file lists

Post by *Horst.Epp »

It looks like the Loadlist command can't handle file lists formatted UTF8 without Bom.
Everything commandline tool creates such lists.
TC complains File not found (46x) followed by 5 dir names and ...
The list is valid and contains no illegal entries for dirs or files.
Other file managers (XYplorer and Free Commander) handle the same list without any problem.
Windows 11 Home x64 Version 23H2 (OS Build 22631.3374)
TC 11.03 x64 / x86
Everything 1.5.0.1371a (x64), Everything Toolbar 1.3.2, Listary Pro 6.3.0.69
QAP 11.6.3.2 x64
User avatar
petermad
Power Member
Power Member
Posts: 14739
Joined: 2003-02-05, 20:24 UTC
Location: Denmark
Contact:

Re: LOADLIST command and UTF8 file lists

Post by *petermad »

Confirmed.

Even if lister recognizes the list file without BOM as UTF-8 then the LOADLIST command does not.
License #524 (1994)
Danish Total Commander Translator
TC 11.03 32+64bit on Win XP 32bit & Win 7, 8.1 & 10 (22H2) 64bit, 'Everything' 1.5.0.1371a
TC 3.50b4 on Android 6 & 13
Try: TC Extended Menus | TC Languagebar | TC Dark Help | PHSM-Calendar
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: LOADLIST command and UTF8 file lists

Post by *ghisler(Author) »

I cannot reproduce this with TC 10. I created a list file via Mark - Save selection to file, then loaded it into notepad and saved as UTF-8 without BOM.
Then tried loading the list via LOADLIST, and it works.

The detection is turned off when
1. The encoding is one of the following:
SHIFTJIS_CHARSET or
GB2312_CHARSET or
HANGEUL_CHARSET or
CHINESEBIG5_CHARSET
and
2. There are no UTF-8 sequences longer than 2 bytes.

In this case the file can be both UTF-8 or 2 byte local encoding.

Also "Load selection from file" does NOT support UTF-8 without BOM.

Please provide a few file names with which it doesn't work.
Author of Total Commander
https://www.ghisler.com
User avatar
Horst.Epp
Power Member
Power Member
Posts: 6450
Joined: 2003-02-06, 17:36 UTC
Location: Germany

Re: LOADLIST command and UTF8 file lists

Post by *Horst.Epp »

Christian, we are not talking about "Load selection from file"
Its about LOADLIST.
The file lists I use are normaly not having a Bom
but also UTF8 with Bom fails for me.
File names or dir names with German characters are the problem in my tests.
Windows 11 Home x64 Version 23H2 (OS Build 22631.3374)
TC 11.03 x64 / x86
Everything 1.5.0.1371a (x64), Everything Toolbar 1.3.2, Listary Pro 6.3.0.69
QAP 11.6.3.2 x64
User avatar
petermad
Power Member
Power Member
Posts: 14739
Joined: 2003-02-05, 20:24 UTC
Location: Denmark
Contact:

Re: LOADLIST command and UTF8 file lists

Post by *petermad »

Sorry, my bad.
I accidently tested LOADSELECTION, which does not always work from UTF-8 list files without BOM.
The LOADLIST command here does work for UTF-8 list files without BOM.

Testfil: https://madsenworld.dk/tcmd/Listtest.zip
License #524 (1994)
Danish Total Commander Translator
TC 11.03 32+64bit on Win XP 32bit & Win 7, 8.1 & 10 (22H2) 64bit, 'Everything' 1.5.0.1371a
TC 3.50b4 on Android 6 & 13
Try: TC Extended Menus | TC Languagebar | TC Dark Help | PHSM-Calendar
User avatar
nsp
Power Member
Power Member
Posts: 1803
Joined: 2005-12-04, 08:39 UTC
Location: Lyon (FRANCE)
Contact:

Re: LOADLIST command and UTF8 file lists

Post by *nsp »

UTF8 without bom is not a regular format for windows OS ! As said by Christian G. LOADLIST is not designed to work with utf8 no BOM files.

This is also the same for UTF16 no bom or BE variant !

Some software produce output without BOM as temp file, this is very convenient to combine multiple results !
You must instruct file processing about the format by bom or by parameter/variant. Automatic decoding can be an ambiguous decoding in some circumstance.

To my knowledge, we do not have loadlist[W|U] command to force UTF8 or UTF16 decoding when no bom is present. The easiest is to add the BOM using intermediate script or ask everything development to add the BOM.

LOADLIST works fine with UTF8/UTF16-LE file with bom and when all files exists !

You could suggest to add LOADLISTW / LOADLISTU ... or have an additional parameter.
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: LOADLIST command and UTF8 file lists

Post by *ghisler(Author) »

2Horst.Epp
Did you actually read and understand my reply? I wrote that LOADLIST does support UTF-8 without BOM, just "Load selection from file" doesn't.

That's why I asked for a list of names where it doesn't work for you.
Author of Total Commander
https://www.ghisler.com
User avatar
Horst.Epp
Power Member
Power Member
Posts: 6450
Joined: 2003-02-06, 17:36 UTC
Location: Germany

Re: LOADLIST command and UTF8 file lists

Post by *Horst.Epp »

ghisler(Author) wrote: 2021-10-05, 09:53 UTC 2Horst.Epp
Did you actually read and understand my reply? I wrote that LOADLIST does support UTF-8 without BOM, just "Load selection from file" doesn't.

That's why I asked for a list of names where it doesn't work for you.
Its a long list and I just try to identify the real problem names.
Windows 11 Home x64 Version 23H2 (OS Build 22631.3374)
TC 11.03 x64 / x86
Everything 1.5.0.1371a (x64), Everything Toolbar 1.3.2, Listary Pro 6.3.0.69
QAP 11.6.3.2 x64
User avatar
nsp
Power Member
Power Member
Posts: 1803
Joined: 2005-12-04, 08:39 UTC
Location: Lyon (FRANCE)
Contact:

Re: LOADLIST command and UTF8 file lists

Post by *nsp »

ghisler(Author) wrote: 2021-10-05, 09:53 UTC That's why I asked for a list of names where it doesn't work for you.
You can download files sent by @PeterMad .
and from the folder where file are extracted create a button like:

Code: Select all

TOTALCMD#BAR#DATA
LOADLIST
%P%N
wcmicons.dll,3



-1
and test fileList with with bom and without bom !
User avatar
petermad
Power Member
Power Member
Posts: 14739
Joined: 2003-02-05, 20:24 UTC
Location: Denmark
Contact:

Re: LOADLIST command and UTF8 file lists

Post by *petermad »

2nsp
You can download files sent by @PeterMad .
and from the folder where file are extracted create a button like:
Just to make it clear: LOADLIST works fine with the files in https://madsenworld.dk/tcmd/Listtest.zip and the listutf8nobom.txt file - it is LOADSELECTION that doesn't select most of those files with listutf8nobom.txt
License #524 (1994)
Danish Total Commander Translator
TC 11.03 32+64bit on Win XP 32bit & Win 7, 8.1 & 10 (22H2) 64bit, 'Everything' 1.5.0.1371a
TC 3.50b4 on Android 6 & 13
Try: TC Extended Menus | TC Languagebar | TC Dark Help | PHSM-Calendar
NotNull
Senior Member
Senior Member
Posts: 266
Joined: 2019-11-25, 20:43 UTC
Location: NL

Re: LOADLIST command and UTF8 file lists

Post by *NotNull »

Just "installed" TC 10 and can confirm that LOADLIST has issues. That is to say:

A button on the buttonbar that runs LOADLIST t:\export.txt works fine, regardless the encoding (ANSI, UTF8, UTF8-BOM).
However, running it from TC's prompt generates the following error, with all mentioned encodings:

Code: Select all

---------------------------
Total Commander
---------------------------
File not found!
LOADLIST

"The system cannot find the file specified."
---------------------------
OK
---------------------------


For reference, t:\export.txt (basically some random files with only lower ASCII characters):

Code: Select all

C:\Tools\WinMerge_2011.007.444_x64\WinMergeU.exe
C:\Tools\UniExtractRC3\UniExtract\bin\RAIU.EXE
C:\Tools\UniExtract\bin\RAIU.EXE
C:\Tools\cygwin64\bin\du.exe


BTW: A command-line parameter to load said file list could be useful. Especially when that file list was generated by an external applicatiuon.
User avatar
petermad
Power Member
Power Member
Posts: 14739
Joined: 2003-02-05, 20:24 UTC
Location: Denmark
Contact:

Re: LOADLIST command and UTF8 file lists

Post by *petermad »

2NotNull
Except for the CD command ALL the commands in the "Commands with parameters" section of totalcmd.inc can NOT be run from the command line - this is not an restriction of the LOADLIST command exclusively
License #524 (1994)
Danish Total Commander Translator
TC 11.03 32+64bit on Win XP 32bit & Win 7, 8.1 & 10 (22H2) 64bit, 'Everything' 1.5.0.1371a
TC 3.50b4 on Android 6 & 13
Try: TC Extended Menus | TC Languagebar | TC Dark Help | PHSM-Calendar
NotNull
Senior Member
Senior Member
Posts: 266
Joined: 2019-11-25, 20:43 UTC
Location: NL

Re: LOADLIST command and UTF8 file lists

Post by *NotNull »

That is quite a restriction ... Do you happen to know why?

Anyway, thanks for clarifying!
User avatar
petermad
Power Member
Power Member
Posts: 14739
Joined: 2003-02-05, 20:24 UTC
Location: Denmark
Contact:

Re: LOADLIST command and UTF8 file lists

Post by *petermad »

2NotNull
That is quite a restriction
You can just make an em_loadlist command with the LOADLIST in your usercmd.ini file for it, and then use that em_loadlist in th command line - for example:

Code: Select all

[em_loadlist]
cmd=LOADLIST
param=%P%N
This command will work in the command line with the file under the cursor.

--------------------

You can also make it work with a filename on the command line with this version:

Code: Select all

[em_loadlist]
cmd=LOADLIST
param=%A
And then make an Alias for the em_loadlist command in your wincmd.ini:

Code: Select all

[Alias]
LOADLIST=em_loadlist
Then you can type: LOADLIST t:\export.txt in the command line as you wanted...
Last edited by petermad on 2021-10-07, 22:02 UTC, edited 1 time in total.
License #524 (1994)
Danish Total Commander Translator
TC 11.03 32+64bit on Win XP 32bit & Win 7, 8.1 & 10 (22H2) 64bit, 'Everything' 1.5.0.1371a
TC 3.50b4 on Android 6 & 13
Try: TC Extended Menus | TC Languagebar | TC Dark Help | PHSM-Calendar
NotNull
Senior Member
Senior Member
Posts: 266
Joined: 2019-11-25, 20:43 UTC
Location: NL

Re: LOADLIST command and UTF8 file lists

Post by *NotNull »

Thanks!

(that is basically what I did as a workaround: creating an em_.. user command. Didn't know about aliases though; handy!)
Post Reply