LOADSEARCH does not support UTF-16 text files w/o BOM

Here you can propose new features, make suggestions etc.

Moderators: white, Hacker, petermad, Stefan2

andry81
Junior Member
Junior Member
Posts: 97
Joined: 2018-11-22, 19:17 UTC

LOADSEARCH does not support UTF-16 text files w/o BOM

Post by *andry81 »

OS: `Windows 7 x64`

I have noticed en error while trying to use an UTF-16LE without BOM text file as a list file for the search dialog window.
1. Is there any solution for that?
2. Can I just run an external script file just before the LOADSEARCH command itself in one command to just convert file without BOM to file with BOM?
3. If all above is not an option, then may be add a simple BOM sequence detection into LOADSEARCH/LOADSELECTION commands logic like, for example, Notepad++ does?
4. If not, then may be add LOADSEARCH_UTF16LE_NOBOM/LOADSELECTION_UTF16LE_NOBOM respective commands?
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: LOADSEARCH does not support UTF-16 text files w/o BOM

Post by *ghisler(Author) »

You could try creating a file with just the BOM (e.g. bom.txt), and then use xcopy command to create file with BOM, e.g.
Command: xcopy
Parameters: /s c:\path\bom.txt %P%N %P%O_bom.%E
Author of Total Commander
https://www.ghisler.com
andry81
Junior Member
Junior Member
Posts: 97
Joined: 2018-11-22, 19:17 UTC

Re: LOADSEARCH does not support UTF-16 text files w/o BOM

Post by *andry81 »

ghisler(Author) wrote: 2020-08-31, 14:09 UTC You could try creating a file with just the BOM (e.g. bom.txt), and then use xcopy command to create file with BOM, e.g.
Command: xcopy
Parameters: /s c:\path\bom.txt %P%N %P%O_bom.%E
I need it in one command to create AND call to LOADSEARCH.
User avatar
petermad
Power Member
Power Member
Posts: 14741
Joined: 2003-02-05, 20:24 UTC
Location: Denmark
Contact:

Re: LOADSEARCH does not support UTF-16 text files w/o BOM

Post by *petermad »

2andry81
Could you show me the saved search you want to load - as it is written in the [Searches] section of your wincmd.ini file - then I might be able to make it for you with combined em_commands.
License #524 (1994)
Danish Total Commander Translator
TC 11.03 32+64bit on Win XP 32bit & Win 7, 8.1 & 10 (22H2) 64bit, 'Everything' 1.5.0.1371a
TC 3.50b4 on Android 6 & 13
Try: TC Extended Menus | TC Languagebar | TC Dark Help | PHSM-Calendar
andry81
Junior Member
Junior Member
Posts: 97
Joined: 2018-11-22, 19:17 UTC

Re: LOADSEARCH does not support UTF-16 text files w/o BOM

Post by *andry81 »

I thought the command dialog does not support mix calls to external scripts and built in commands, but anyway:
saveload_search_in_utf16le_slot_01_SearchFor=*.*
saveload_search_in_utf16le_slot_01_SearchIn=@...path_to_file...\01.utf-16le.lst
saveload_search_in_utf16le_slot_01_SearchText=
saveload_search_in_utf16le_slot_01_SearchFlags=0|103002010021|||||||||0000|0||
User avatar
petermad
Power Member
Power Member
Posts: 14741
Joined: 2003-02-05, 20:24 UTC
Location: Denmark
Contact:

Re: LOADSEARCH does not support UTF-16 text files w/o BOM

Post by *petermad »

I thought the command dialog does not support mix calls to external scripts and built in commands, but anyway:
You have to assign the scripts to em_commands that you can then mix:

Make 3 em_commands in your usercmd.ini file like this:

Code: Select all

[em_addbom]
cmd=%COMSPEC% /C
param=copy /Y /B "%%COMMANDER_PATH%%\bom16le.txt" + %P%N "%P%O_bom.%E" && copy "%P%O_bom.%E" %P%N && del "%P%O_bom.%E"
iconic=1

[em_load_search]
cmd=LOADSEARCH saveload_search_in_utf16le_slot_01

[em_addbom_load_search]
cmd=em_addbom, em_load_search
Now you can use the em_addbom_load_search command in a button or assign it to a keyboard shortcut.
You need the bom16le.txt file placed in your Total Commander folder - you can get the file here: https://madsenworld.dk/tcmd/boms.zip

With this solution you have to place TC's cursor on the 01.utf-16le.lst file - if you always use the file in the same location you can change the em_addbom command to for example:

Code: Select all

[em_addbom]
cmd=%COMSPEC% /C
param=copy /Y /B "%%COMMANDER_PATH%%\bom16le.txt" + "c:\test\01.utf-16le.lst" "c:\test\01.utf-16le_bom.lst" && copy "c:\test\01.utf-16le_bom.lst" "c:\test\01.utf-16le.lst" && del "c:\test\01.utf-16le_bom.lst"
iconic=1
replace c:\test\ with your path.

If you only want to use this from a button, then you can skip em_addbom_load_search and just use em_addbom, em_load_search directly in the buttons Command field.
License #524 (1994)
Danish Total Commander Translator
TC 11.03 32+64bit on Win XP 32bit & Win 7, 8.1 & 10 (22H2) 64bit, 'Everything' 1.5.0.1371a
TC 3.50b4 on Android 6 & 13
Try: TC Extended Menus | TC Languagebar | TC Dark Help | PHSM-Calendar
andry81
Junior Member
Junior Member
Posts: 97
Joined: 2018-11-22, 19:17 UTC

Re: LOADSEARCH does not support UTF-16 text files w/o BOM

Post by *andry81 »

Thx for the example, but it is quite not enough. There is a need to check the input file before the copy if it already has a bom sequence.
User avatar
petermad
Power Member
Power Member
Posts: 14741
Joined: 2003-02-05, 20:24 UTC
Location: Denmark
Contact:

Re: LOADSEARCH does not support UTF-16 text files w/o BOM

Post by *petermad »

2andry81
OK - try this:

make a file bomfix.cmd and place it in your total Commander folder::

Code: Select all

@echo off
setlocal EnableExtensions DisableDelayedExpansion

rem // Define constants here:
set "_FILE=%~1" & rem // (provide file via the first command line argument)

rem // Check whether a dedicated file is given (so no wild-cards):
2> nul >&2 (< "%_FILE%" set /P ="" & ver) || (
    rem // The file does not exist:
    >&2 echo The file could not be found, hence there is no encoding!
    exit /B 255
)

rem // Determine the file size:
set "SIZE=" & for %%F in ("%_FILE%") do set "SIZE=%%~zF"
if not defined SIZE (
    rem // The file does not exist:
    >&2 echo The file could not be found, hence there is no encoding!
    exit /B 255
)
if %SIZE% EQU 0 (
    rem // The file is empty:
    >&2 echo The file is empty, make an empty file with BOM
    goto :CONT
)

rem // Store current code page to be able to restore it finally:
for /F "tokens=2 delims=:" %%C in ('chcp') do set /A "$CP=%%C"
rem /* Change to code page 437 (original IBM PC or DOS code page) temporarily;
rem    this is necessary for extended characters not to be converted: */
> nul chcp 437


rem /* Build Byte Order Marks (BOMs) for UTF-16-encoded text (Little Endian and Big Endian)
rem    and for UTF-8-encoded text: */
for /F "tokens=1-3" %%A in ('
    forfiles /P "%~dp0." /M "%~nx0" /C "cmd /C echo 0xFF0xFE 0xFE0xFF 0xEF0xBB0xBF"
') do set "$LE=%%A" & set "$BE=%%B" & set "$U8=%%C"

rem /* Reset line string variable, then store first line string (1023 bytes at most);
rem    in contrast to `for /F`, this does not skip over blank lines: */
< "%_FILE%" (set "LINE=" & set /P LINE="")
rem // Check whether the first line of the file begins with any of the BOMs:
if not "%LINE:~,2%"=="%$LE%" if not "%LINE:~,2%"=="%$BE%" if not "%LINE:~,3%"=="%$U8%" goto :CONT
rem /* One of the BOMs has been encountered;
rem    restore the initial code page prior to termination: */
> nul chcp %$CP%
>&2 echo BOM encountered in first line
exit /B 1

rem // This point is reached in case the file does not have a BOM:
:CONT
rem // Restore the initial code page prior to termination:
> nul chcp %$CP%
copy /Y /B "%COMMANDER_PATH%\bom16le.txt" + "%1" "%1bom" && copy "%1bom" "%1" && del "%1bom"
echo BOM added to first line.

endlocal
exit /B 0

Use there commands in your usercmd.ini file:

Code: Select all

[em_addbom]
cmd=%COMMANDER_PATH%\bomfix.cmd
param=%P%N
iconic=1

[em_load_search]
cmd=LOADSEARCH saveload_search_in_utf16le_slot_01

[em_addbom_load_search]
cmd=em_addbom, em_load_search
You can replace %P%N in the em_addbom command with a fixed path\filename to your .lst file.

bomfix.cmd actually tests for the presence of LE bom, BE bom and UFT8 bom, but only sets LE bom (using the bom16le.txt file).


Files can be downloaded at https://madsenworld.dk/tcmd/boms2.zip

.
License #524 (1994)
Danish Total Commander Translator
TC 11.03 32+64bit on Win XP 32bit & Win 7, 8.1 & 10 (22H2) 64bit, 'Everything' 1.5.0.1371a
TC 3.50b4 on Android 6 & 13
Try: TC Extended Menus | TC Languagebar | TC Dark Help | PHSM-Calendar
andry81
Junior Member
Junior Member
Posts: 97
Joined: 2018-11-22, 19:17 UTC

Re: LOADSEARCH does not support UTF-16 text files w/o BOM

Post by *andry81 »

andry81
Junior Member
Junior Member
Posts: 97
Joined: 2018-11-22, 19:17 UTC

Re: LOADSEARCH does not support UTF-16 text files w/o BOM

Post by *andry81 »

[em_addbom]
cmd=%COMMANDER_PATH%\bomfix.vbs

[em_load_search]
cmd=LOADSEARCH saveload_search_in_utf16le_slot_01

[em_addbom_load_search]
cmd=em_addbom, em_load_search
I've found an issue: the em_load_search for some reason does not wait completion of the em_addbom, so selects files from the previously generated list file.
User avatar
petermad
Power Member
Power Member
Posts: 14741
Joined: 2003-02-05, 20:24 UTC
Location: Denmark
Contact:

Re: LOADSEARCH does not support UTF-16 text files w/o BOM

Post by *petermad »

2andry81
've found an issue: the em_load_search for some reason does not wait completion of the em_addbom
Have you tried using TC's cm_wait command - like:

Code: Select all

[em_addbom_load_search]
cmd=em_addbom, cm_wait 1000, em_load_search
Maybe that could solve it.
License #524 (1994)
Danish Total Commander Translator
TC 11.03 32+64bit on Win XP 32bit & Win 7, 8.1 & 10 (22H2) 64bit, 'Everything' 1.5.0.1371a
TC 3.50b4 on Android 6 & 13
Try: TC Extended Menus | TC Languagebar | TC Dark Help | PHSM-Calendar
andry81
Junior Member
Junior Member
Posts: 97
Joined: 2018-11-22, 19:17 UTC

Re: LOADSEARCH does not support UTF-16 text files w/o BOM

Post by *andry81 »

Have you tried using TC's cm_wait command - like:
Yes, but that is kind of ugly workaround.
User avatar
nsp
Power Member
Power Member
Posts: 1804
Joined: 2005-12-04, 08:39 UTC
Location: Lyon (FRANCE)
Contact:

Re: LOADSEARCH does not support UTF-16 text files w/o BOM

Post by *nsp »

You should call to TC yourself after converting your file using tcfs2 tem or better convert the file once saved before using it in TC.You will have same timing issue

Code: Select all

tcfs2 /ef tem(`em_loadsearch "search to load"`)
tcfs2 /ef tem(em_load_search)
  • em_loadsearch is LOADSEARCH with parameter %A
  • em_load_search is LOADSEARCH with predefined parameter
--edited--
andry81
Junior Member
Junior Member
Posts: 97
Joined: 2018-11-22, 19:17 UTC

Re: LOADSEARCH does not support UTF-16 text files w/o BOM

Post by *andry81 »

You should call to TC yourself after converting your file
Does the tcfs2 active the same instance or create a new one? It looks strange, how it does detect the correct instance from multiple instances?
Post Reply