New SFTP plugin available now

[JOUBE]

Hi,
I hope, I do not touch any rejected topic here since I did not manually read the 90 pages of discussion. But I did search and found only similar topics concerning the main application and the included FTP-functionality, not the SFTP-plugin yet.

So I would like to request some extra columns for the SFTP-plugin. - For me only "owner" and "group" would be sufficient.
It is very cumbersome to [Alt]+[Enter] on every single file - and start again from the beginning if I just forgot the first owners already!

Really searched? On page 90 of this thread - on which you have posted(!) -, you can find this: viewtopic.php?p=429570#p429570:

Please add columns for user and group in long view of the sftp plugin

[lbug]

Oh sorry!

I had searched using startpage with search string like "TotalCommander SFTP show owner in column" and similar.
But I got only results like
https://ghisler.ch/board/viewtopic.php?t=23379
viewtopic.php?t=18370
and maybe others I have not open any more.

Even to find this thread I had to manually browse the forum.
And yes, I read the opening post as well as skimmed through page 90 - sadly overlooking your keywords "user" and "group"!

But at least this does show the demand for such a functionality.

[Stefan2]

Oh sorry!

Please don't worry. At least you have searched.


You may also utilize Google to search the forum, f.ex.: SFTP owner column site:ghisler.ch

That finds this request even on "- Page 32 -" >> viewtopic.php?p=192297#p192297 including an answer



Use the features of google search to find f.ex. by age (last Year) and such.

 

[petermad]

Generally the board's search function sucks, but this search: https://ghisler.ch/board/search.php?keywords=sftp+column+user+group with 4 keywords does the job and currently only gives 3 hits (including tghis post).

[JOUBE]

But at least this does show the demand for such a functionality.

The last three months here in the forum were characterized by the preparations for the release of TC Version 11. It started with Tc11 Beta 1 on May 3rd, 2023. In the meantime, 11 betas and 3 release candidates have appeared and numerous new functions have been tested. And TC11 is imminent.

I explain this so that you understand why ghisler(author) didn't have the time and quiet to turn to the SFTP plugin. That's why I wrote a summary suggestion in between, which you are welcome to comment on and add to if necessary. I also find that there is a need for the SFTP plugin: When Tc11 is released, next step should be a look at the SFTP plugin

Joube

[lbug]

No problem! No problem at all!

I do know how much work goes into all this development and I did not want to stress somebody.
I really just thought this specific topic was not mentioned before in context of the SFTP-plugin ... according to my insufficient search.

[ghisler(Author)]

Unfortunately reading a directory via SFTP only returns numerical UID and GID values. Only the stat command used on Alt+Enter returns both the numerical values and the user/group name.

[JOUBE]

Unfortunately reading a directory via SFTP only returns numerical UID and GID values. Only the stat command used on Alt+Enter returns both the numerical values and the user/group name.

This is not true. If you have enabled detailedlog=1 in the settings of a connection and have a look at the logfile while a directory is listed, you will see user and group.

Joube

[ghisler(Author)]

That's odd, the LIBSSH2_SFTP_ATTRIBUTES structure returned by libssh2_sftp_readdir_ex only contains the numeric uid and gid.
From file libssh2_sftp.h:

Code: Select all

struct _LIBSSH2_SFTP_ATTRIBUTES {
    /* If flags & ATTR_* bit is set, then the value in this struct will be
     * meaningful Otherwise it should be ignored
     */
    unsigned long flags;

    libssh2_uint64_t filesize;
    unsigned long uid, gid;
    unsigned long permissions;
    unsigned long atime, mtime;
};

The line you see in the log seems to be raw data from the server which still needs to be interpreted. It may differ between servers, especially if the server isn't Unix based.

[lbug]

Just as a hint:
Maybe you could look at the source of WinSCP (open source) - how they are getting this piece of information?

Though, I do not know what libraries they are using.
But it should be possible in an efficient way. - Using WinSCP I can easily open directories which do contain thousands of files (with user and group names already on every entry) and it does not take any noticeable time longer to load than the SFTP-plugin.

[JOUBE]

2ghisler(author)
This - which you called "raw data" - come from the servers. I think it comes from OpenSSH *), because windows10/11 send exact the same. Because you use findfirst/findnext to get the data (within your sftpfunc.cpp: SftpFindFirstFileW() and SftpFindNextFileW() and - (if detailedlog is true) at line 3214 - ShowStatus(completeline); ) and the answer of the servers are allways unsorted and you store this unsorted data to tcftp.log. I can simulate this unsorted data by sending "ls -laU" where -U means unsorted, and the sorting VIA SHELL in this case is exact the same 'order' with the ssh/scp/sfpt:finsfirst/findnext. I will send you some log data in next post or via email after i have anonymized them.

Joube

*) and other if they have implmented version 2.xx

Edit typo: 'oder' -> 'order', same -> some, date -> data

[JOUBE]

Just as a hint:
Maybe you could look at the source of WinSCP (open source) - how they are getting this piece of information?

This is not necessary as everything is there in his sources and in the log and ghisler(author) will see it immediately.

Though, I do not know what libraries they are using.
But it should be possible in an efficient way. - Using WinSCP I can easily open directories which do contain thousands of files (with user and group names already on every entry) and it does not take any noticeable time longer to load than the SFTP-plugin.

Not so complicated please, everything is already there and arrives "for free" via ssh.

[JOUBE]

Part of tcftp.log (Blank lines are added by me):

Code: Select all

CONNECT \Debian 12
Verzeichnis einlesen: \home\user1\folder1
-rw-r--r--    1 user1    user1      369880 Jul  7  2020 rrd1.rrd
-rwxr-xr-x    1 user1    user1         586 Jun 11  2022 script1.sh
-rw-r--r--    1 user1    user1        2840 Feb 23 14:12 script2.sh
-rwxr-xr-x    1 user1    user1         796 May 11  2020 script3.sh
...
-rw-r--r--    1 user1    user1        4275 Jul  9  2020 script4.sh


Verzeichnis einlesen: \home\user1
-rw-r--r--    1 user1    user1         807 Feb 19  2020 .profile
drwxr-xr-x    6 user1    user1        4096 Jun 25  2020 folder1
drwxr--r--    2 user1    user1        4096 Sep  7  2021 .theinfo
...
drwxr-xr-x    3 root     root         4096 Jun  1 06:20 ..


Verzeichnis einlesen: \home
...
drwxr-xr-x    3 root     root         4096 Jun  1 06:20 .
drwxr-xr-x   19 root     root         4096 Jul  6 04:54 ..


Verzeichnis einlesen: \
drwxr-xr-x   28 root     root          840 Jul 31 12:28 run
...
drwxr-xr-x    2 root     root         4096 Feb 19  2020 srv
drwxr-xr-x   19 root     root         4096 Jul  6 04:54 ..


Verzeichnis einlesen: \etc
drwxr-xr-x  151 root     root        12288 Jul 31 07:53 .
drwxr-xr-x   19 root     root         4096 Jul  6 04:54 ..
drwxr-xr-x    4 root     root         4096 Mar 27  2022 ghostscript
drwxrwxr-x    2 fxclient root         4096 Mar 30 10:49 fxclient
...
-rw-r-----    1 root     shadow       1423 May 31 12:52 shadow
drwxr-xr-x    2 root     root         4096 Jun  2 04:54 PackageKit
lrwxrwxrwx    1 root     root           21 Jul 14 18:00 os-release
DISCONNECT \Debian 12
========================



CONNECT \Windows 10
Verzeichnis einlesen: \C:\Dir1
drwx******    1 -        -               0 Oct 26  2022 - # i6 c
drwx******    1 -        -               0 Jun 25 18:13 Dir2
drwx******    1 -        -            8192 Apr 14 11:15 Dir3
...
drwx******    1 -        -          393216 Jul 31 10:07 Dir4


Verzeichnis einlesen: \C:\Dir1\Dir4
-rw-******    1 -        -            1573 Jul 27 09:39 batch1.ahk
-rwx******    1 -        -            2607 Jul 27 09:39 batch2.cmd
...
-rwx******    1 -        -              58 Oct 18  2018 batch3.cmd
-rwx******    1 -        -              58 Nov 30  2017 batch4.cmd
-rwx******    1 -        -            6058 Feb  6  2021 batch5.cmd
DISCONNECT \Windows 10
========================





CONNECT \Windows 11
Verzeichnis einlesen: \C:\Dir1
drwx******    1 -        -               0 Sep  7  2022 Dir2
drwx******    1 -        -               0 Feb 28 15:52 Dir3
drwx******    1 -        -            8192 Dec  1  2021 Dir5
drwx******    1 -        -           65536 Jul  6 12:40 Txt
drwx******    1 -        -          393216 Jul 27 09:53 Dir4


Verzeichnis einlesen: \C:\Dir1\Dir2
drwx******    1 -        -               0 Sep  7  2022 Dri2a
DISCONNECT \Windows 10
========================





CONNECT \Fedora38
Verzeichnis einlesen: \home\user1
drwxr-xr-x    4 user1    user1        4096 May 21 11:57 .local
drwxrwxrwx    8 user1    user1        4096 Jul 21 10:52 folder1
drwxr-xr-x    7 user1    user1        4096 Apr 21 11:56 folder2
drwxr-xr-x    3 root     root         4096 Jun 11 12:22 ..
drwxr-xr-x    2 user1    user1        4096 May 21 11:40 folder3
drw-r--r--    1 user1    user1         141 Feb  6 01:00 .bash_profile


Verzeichnis einlesen: \home
dr-xr-xr-x   19 root     root         4096 Jun 17 12:16 ..
drwxr-xr-x    3 root     root         4096 Jun 11 12:22 .
drwx------   23 user1    user1        4096 Jul 31 06:11 user1


Verzeichnis einlesen: \
dr-xr-xr-x   13 root     root            0 Jul  1 23:30 sys
dr-xr-xr-x   19 root     root         4096 Jun 17 12:16 ..
[...]
rwxrwxrwx    1 root     root            8 Jan 19  2023 sbin


Verzeichnis einlesen: \etc
drwxr-xr-x  137 root     root        12288 Jul 31 05:15 .
drwxr-xr-x    2 root     root         4096 Apr 11 02:00 gcrypt
drwxr-xr-x    2 root     root         4096 Apr 13 23:45 xscreensaver
drwxr-xr-x    4 root     lp           4096 Jul  1 05:17 cups
drwxr-xr-x    2 root     root         4096 Apr 13 23:45 request-key.d
drwxr-xr-x    3 root     root         4096 Jun 29 02:00 gss
-rw-r--r--    1 root     root         1968 Jan 19  2023 mail.rc
drwxr-xr-x   14 root     root         4096 Jun 16 09:19 xdg
-rw-r--r--    1 root     root         3223 Jun 21 02:00 rsyslog.conf
-rw-r--r--    1 root     root           94 Jan 19  2023 GREP_COLORS
drwxr-xr-x    2 root     root         4096 May 21 11:48 modprobe.d
drwxr-xr-x    2 root     root         4096 Jan 19  2023 depmod.d
[...]
-r--r-----    1 root     root         4402 Jun  1 10:33 sudoers
DISCONNECT \Fedora 38
========================

Joube

[JOUBE]

This is what you get: https://www.libssh2.org/libssh2_sftp_readdir_ex.html

Code: Select all

int
libssh2_sftp_readdir_ex(LIBSSH2_SFTP_HANDLE *handle,
                        char *buffer, size_t buffer_maxlen,
                        char *longentry, size_t longentry_maxlen,
                        LIBSSH2_SFTP_ATTRIBUTES *attrs);

libssh2_sftp_readdir_ex :
longentry - is a pointer to a pre-allocated buffer of at least longentry_maxlen bytes to read data into. The format of the `longname' field is unspecified by SFTP protocol. It MUST be suitable for use in the output of a directory listing command (in fact, the recommended operation for a directory listing command is to simply display this data). So:

The line you see in the log seems to be raw data from the server which still needs to be interpreted.

Yes. But it is very easy to do that. And it is needed to do that for a full and usefull sftp implementation. If you do not want to implement columns for user and group itself, than you have to implement in long view a column with this *longentry completely.

It may differ between servers, especially if the server isn't Unix based.

(Unix based = ?, Linux = OpenSSH based, Windows based = Openssh based (pre w10 = ?), Mac = ?) I will check that (with some other ssh Server). Version 2.

[Dalai]

(Unix based = ?, Linux = OpenSSH based, Windows based = Openssh based (pre w10 = ?), Mac = ?) I will check that (with some other ssh Server). Version 2.

Who says there's only OpenSSH for Linux? Dropbear is another SSH server for Linux, a very lightweight one that is primarily used in embedded systems, as well as in the x86 distro named Fli4l. And I'm pretty sure there are other SSH servers for Linux. And don't forget FreeBSD, NetBSD, HP-UX, Sun and other systems with a different architecture and software packages. All of them could behave differently, deliver different outputs and so on.

Regards
Dalai