LOADSEARCH does not support UTF-16 text files w/o BOM

[andry81]

OS: `Windows 7 x64`

I have noticed en error while trying to use an UTF-16LE without BOM text file as a list file for the search dialog window.
1. Is there any solution for that?
2. Can I just run an external script file just before the LOADSEARCH command itself in one command to just convert file without BOM to file with BOM?
3. If all above is not an option, then may be add a simple BOM sequence detection into LOADSEARCH/LOADSELECTION commands logic like, for example, Notepad++ does?
4. If not, then may be add LOADSEARCH_UTF16LE_NOBOM/LOADSELECTION_UTF16LE_NOBOM respective commands?

[ghisler(Author)]

You could try creating a file with just the BOM (e.g. bom.txt), and then use xcopy command to create file with BOM, e.g.
Command: xcopy
Parameters: /s c:\path\bom.txt %P%N %P%O_bom.%E

[andry81]

You could try creating a file with just the BOM (e.g. bom.txt), and then use xcopy command to create file with BOM, e.g.
Command: xcopy
Parameters: /s c:\path\bom.txt %P%N %P%O_bom.%E

I need it in one command to create AND call to LOADSEARCH.

[petermad]

2andry81
Could you show me the saved search you want to load - as it is written in the [Searches] section of your wincmd.ini file - then I might be able to make it for you with combined em_commands.

[andry81]

I thought the command dialog does not support mix calls to external scripts and built in commands, but anyway:

saveload_search_in_utf16le_slot_01_SearchFor=*.*
saveload_search_in_utf16le_slot_01_SearchIn=@...path_to_file...\01.utf-16le.lst
saveload_search_in_utf16le_slot_01_SearchText=
saveload_search_in_utf16le_slot_01_SearchFlags=0|103002010021|||||||||0000|0||

[petermad]

I thought the command dialog does not support mix calls to external scripts and built in commands, but anyway:

You have to assign the scripts to em_commands that you can then mix:

Make 3 em_commands in your usercmd.ini file like this:

Code: Select all

[em_addbom]
cmd=%COMSPEC% /C
param=copy /Y /B "%%COMMANDER_PATH%%\bom16le.txt" + %P%N "%P%O_bom.%E" && copy "%P%O_bom.%E" %P%N && del "%P%O_bom.%E"
iconic=1

[em_load_search]
cmd=LOADSEARCH saveload_search_in_utf16le_slot_01

[em_addbom_load_search]
cmd=em_addbom, em_load_search

Now you can use the em_addbom_load_search command in a button or assign it to a keyboard shortcut.
You need the bom16le.txt file placed in your Total Commander folder - you can get the file here: https://madsenworld.dk/tcmd/boms.zip

With this solution you have to place TC's cursor on the 01.utf-16le.lst file - if you always use the file in the same location you can change the em_addbom command to for example:

Code: Select all

[em_addbom]
cmd=%COMSPEC% /C
param=copy /Y /B "%%COMMANDER_PATH%%\bom16le.txt" + "c:\test\01.utf-16le.lst" "c:\test\01.utf-16le_bom.lst" && copy "c:\test\01.utf-16le_bom.lst" "c:\test\01.utf-16le.lst" && del "c:\test\01.utf-16le_bom.lst"
iconic=1

replace c:\test\ with your path.

If you only want to use this from a button, then you can skip em_addbom_load_search and just use em_addbom, em_load_search directly in the buttons Command field.

[andry81]

Thx for the example, but it is quite not enough. There is a need to check the input file before the copy if it already has a bom sequence.

[petermad]

2andry81
OK - try this:

make a file bomfix.cmd and place it in your total Commander folder::

Code: Select all

@echo off
setlocal EnableExtensions DisableDelayedExpansion

rem // Define constants here:
set "_FILE=%~1" & rem // (provide file via the first command line argument)

rem // Check whether a dedicated file is given (so no wild-cards):
2> nul >&2 (< "%_FILE%" set /P ="" & ver) || (
    rem // The file does not exist:
    >&2 echo The file could not be found, hence there is no encoding!
    exit /B 255
)

rem // Determine the file size:
set "SIZE=" & for %%F in ("%_FILE%") do set "SIZE=%%~zF"
if not defined SIZE (
    rem // The file does not exist:
    >&2 echo The file could not be found, hence there is no encoding!
    exit /B 255
)
if %SIZE% EQU 0 (
    rem // The file is empty:
    >&2 echo The file is empty, make an empty file with BOM
    goto :CONT
)

rem // Store current code page to be able to restore it finally:
for /F "tokens=2 delims=:" %%C in ('chcp') do set /A "$CP=%%C"
rem /* Change to code page 437 (original IBM PC or DOS code page) temporarily;
rem    this is necessary for extended characters not to be converted: */
> nul chcp 437


rem /* Build Byte Order Marks (BOMs) for UTF-16-encoded text (Little Endian and Big Endian)
rem    and for UTF-8-encoded text: */
for /F "tokens=1-3" %%A in ('
    forfiles /P "%~dp0." /M "%~nx0" /C "cmd /C echo 0xFF0xFE 0xFE0xFF 0xEF0xBB0xBF"
') do set "$LE=%%A" & set "$BE=%%B" & set "$U8=%%C"

rem /* Reset line string variable, then store first line string (1023 bytes at most);
rem    in contrast to `for /F`, this does not skip over blank lines: */
< "%_FILE%" (set "LINE=" & set /P LINE="")
rem // Check whether the first line of the file begins with any of the BOMs:
if not "%LINE:~,2%"=="%$LE%" if not "%LINE:~,2%"=="%$BE%" if not "%LINE:~,3%"=="%$U8%" goto :CONT
rem /* One of the BOMs has been encountered;
rem    restore the initial code page prior to termination: */
> nul chcp %$CP%
>&2 echo BOM encountered in first line
exit /B 1

rem // This point is reached in case the file does not have a BOM:
:CONT
rem // Restore the initial code page prior to termination:
> nul chcp %$CP%
copy /Y /B "%COMMANDER_PATH%\bom16le.txt" + "%1" "%1bom" && copy "%1bom" "%1" && del "%1bom"
echo BOM added to first line.

endlocal
exit /B 0

Use there commands in your usercmd.ini file:

Code: Select all

[em_addbom]
cmd=%COMMANDER_PATH%\bomfix.cmd
param=%P%N
iconic=1

[em_load_search]
cmd=LOADSEARCH saveload_search_in_utf16le_slot_01

[em_addbom_load_search]
cmd=em_addbom, em_load_search

You can replace %P%N in the em_addbom command with a fixed path\filename to your .lst file.

bomfix.cmd actually tests for the presence of LE bom, BE bom and UFT8 bom, but only sets LE bom (using the bom16le.txt file).


Files can be downloaded at https://madsenworld.dk/tcmd/boms2.zip

.

[andry81]

Thx for the effort, but I've already made my own conversion script:
https://sourceforge.net/p/contools/contools/HEAD/tree/trunk/Scripts/Tools/encoding/prepend_bom_to_utf_file.bat

[andry81]

[em_addbom]
cmd=%COMMANDER_PATH%\bomfix.vbs

[em_load_search]
cmd=LOADSEARCH saveload_search_in_utf16le_slot_01

[em_addbom_load_search]
cmd=em_addbom, em_load_search

I've found an issue: the em_load_search for some reason does not wait completion of the em_addbom, so selects files from the previously generated list file.

[andry81]

Opened new topic here: TC9.51: Sequence of user defined commands with external script does execute simultaneously instead of sequentially

[petermad]

2andry81

've found an issue: the em_load_search for some reason does not wait completion of the em_addbom

Have you tried using TC's cm_wait command - like:

Code: Select all

[em_addbom_load_search]
cmd=em_addbom, cm_wait 1000, em_load_search

Maybe that could solve it.

[andry81]

Have you tried using TC's cm_wait command - like:

Yes, but that is kind of ugly workaround.

[nsp]

You should call to TC yourself after converting your file using tcfs2 tem or better convert the file once saved before using it in TC.You will have same timing issue

Code: Select all

tcfs2 /ef tem(`em_loadsearch "search to load"`)
tcfs2 /ef tem(em_load_search)

• em_loadsearch is LOADSEARCH with parameter %A

• em_load_search is LOADSEARCH with predefined parameter

--edited--

[andry81]

You should call to TC yourself after converting your file

Does the tcfs2 active the same instance or create a new one? It looks strange, how it does detect the correct instance from multiple instances?