[SOLVED] Lister Ctrl+C (Copy to Clipboard) has issues with Unix-style line breaks/newline

[gdpr deleted 6]

Tested and observed on: Total Commander 9.51RC5, 9.22a; both 32-bit and 64-bit versions

(Update: This issue is present since at least version 7.57a - the oldest version i have tested. I would not be surprised if there is already a report about this issue somewhere in the forum, unfortunately i wasn't able to find one using both the forum search and Google - with the site: operator - using various combinations of different search terms )


When a text selection in Lister ends with a Unix-style line break ("\n"), Lister ignores this "\n" at the end of the selction or converts it into a space character (0x20).

This only happens to a Unix-style line-break at the very end of the selection. Any other Unix-style line breaks within the selection do not suffer from this issue. (Windows-style line breaks "\r\n" are unaffected by this issue.)


How to reproduce:

1. Start up TC with a fresh (or without) wincmd.ini.
2. Display a text file with Unix-style line breaks (using single \n as line-breaks).
3. Select two or more text lines. Make sure that the selection ends with the Unix-style line break of the last selected line.
4. Ctrl+C.
5. Use a Clipboard monitor, some text editor, or write your own program/script to inspect the original clipboard content.

Expected result:

Selected text will be copied as-is to the clipboard, including the last Unix-style line break at the end of the selection.


Observed results:

• When Lister is in "Text Only" view mode: Lister converts the last Unix-style line break at the end of the selection into a space character (0x20). Any other Unix-style line breaks within the selection remain as-is.

• When Lister is in "UTF-8" view mode: Lister simply discards the last Unix-style line break at the end of the selection when copying the selection to the clipboard. Any other Unix-style line breaks within the selection remain as-is.

• When Lister is in "Unicode" view mode: (showing a Unicode-encoded text file with Unix-style line breaks) Like "UTF-8" view mode, Lister simply discards the last Unix-style line break at the end of the selection when copying the selection to the clipboard. Any other Unix-style line breaks within the selection remain as-is.

[gdpr deleted 6]

In case someone does not have a Clipboard monitor at hand or doesn't trust their text editor of choice to not auto-convert clipboard content, here is a crude Powershell script that displays the raw text content (without any conversion) character by character.

Attention! For this script to work, Powershell must be started with the -sta option (Single thread apartment mode). Otherwise, the Clipboard class used in the script won't be able to access the clipboard content.

Code: Select all

function ShowClipboardText()
{
    add-type -an system.windows.forms
    $clipData = [System.Windows.Forms.Clipboard]::GetDataObject()
    foreach ($c in $clipData.GetData("UnicodeText", $False).ToCharArray()) { "0x{0:X2} ({1})" -f [int]$c,$c }
}

The script prints both the hex values of the characters in the clipboard and the characters themselves. Thus, when it prints a newline character (\n) in the console, the console actually executes the newline. Keep that in mind when looking at the output of the script...

[Dalai]

Confirmed. However, other programs also have issues copying the correct characters. The SynWrite plugin converts the line-breaks into CRLF, HxD converts them into 0x2E. Only Notepad++ - and curiously enough Notepad - seem to do it right.

Regards
Dalai

[gdpr deleted 6]

The SynWrite plugin converts the line-breaks into CRLF

Converting a Unix-style LF into a Windows-style CR+LF is tolerable, as the nature of the thing (= "line break") remains the same.

HxD converts them into 0x2E.

No, it doesn't.
It only shows control characters as dots (0x2E) in its UI, but when selecting and copying such characters, they are correctly copied to the clipboard. (At least the 2.1.0.0 and - since i just checked for an update right now -- the 2.4.0.0 versions i am using do it this way)

Notepad

Yeah. After many, many, many years MS finally improved Notepad's line ending handling in 2018.

[Dalai]

No, it doesn't.

Version 1.7.x does. (I didn't have the need to upgrade to version 2.x yet.)

Yeah. After many, many, many years MS finally improved Notepad's line ending handling in 2018.

It's not required to use Notepad on Win10. It also works on Win7, although the selection of LF line-breaks is a little bit cumbersome.

Regards
Dalai

[ghisler(Author)]

I remember vaguely that I had to change that last line break because pasting such text to certain programs caused them to crash. Unfortunately I cannot find this documented anywhere (it was decades ago), but I'm reluctant to change it now...

[gdpr deleted 6]

Well, if Unix-style line endings was/is a concern, why is it only a concern at the end of the selection, but not within the selection?
Anyway, more importantly, why translating a Unix-style line ending into a space or ignoring it, instead of translating into a CR+LF? And why is there an inconsistent behavior depending on view mode to begin with?

It doesn't really make a whole lot of sense...

When some program crashes if TC's lister puts stuff into the clipboard, well it would also crash if other programs (such as Notepad++) would put the "stuff" into the clipboard, no? Why trying to bodge TC in an attempt to "fix" such a program, which still would remain broken and crash-prone unless you constrain yourself to use only Total Commander's lister and would never use no other program to copy text from a text file into the clipboard? I know (i am sorry), that was a situation decades ago, so it is pointless to argue about deeds done. But the thing to take away here is that this bodge in TC's lister effectively hampers a tremendously useful and tremendously basic feature (copying text to clipboard), decades later...

I am not asking nor insisting that you need to fix it now. I am not trying to be pushy here. But at the heart of thing, what the lister currently does with Unix-style line endings at the end of a selection cannot be really explained as some kind of feature...

[gdpr deleted 6]

11.11.20 Fixed: Lister: Copy text from Unix text file -> do not convert trailing 0x10 to space (32/64)

Awesome. It has been fixed, but only partially in TC 10.00b1a.

In "Text only" mode all is now good, with a trailing Unix-style line break in the selection now being preserved as line break (whether Unix-style or not) when copying into the clipboard. Yay!

However, in UTF-8 and Unicode modes, a trailing Unix-style line break in the selection is still being involuntarily discarded

[ghisler(Author)]

I will check it, thanks.

[gdpr deleted 6]

UTF-8 and Unicode mode confirmed fixed in TC 10.00b2 x86 and x64. Thanks!