FTP: Support for MLSD

[tumasch]

It would be very (!) nice if there would be support for the MLSD command. This would replace (or better enhance) the LIST command.

Example:

LIST gives (Filezilla Server):

Code: Select all

-rwxr-xr-x 1 ftp ftp          73802 Sep 28  2009 ab.exe
-rwxr-xr-x 1 ftp ftp          41051 Sep 28  2009 ApacheMonitor.exe
-rw-r--r-- 1 ftp ftp          28771 Sep 28  2009 apr_dbd_mysql-1.dll
-rw-r--r-- 1 ftp ftp          28770 Sep 28  2009 apr_dbd_odbc-1.dll
-rwxr-xr-x 1 ftp ftp          24645 Sep 28  2009 httpd.exe
-rwxr-xr-x 1 ftp ftp          53329 Sep 28  2009 httxt2dbm.exe
drwxr-xr-x 1 ftp ftp              0 Dec 21 17:59 iconv
-rw-r--r-- 1 ftp ftp         135254 Sep 28  2009 libapr-1.dll
-rw-r--r-- 1 ftp ftp          36958 Sep 28  2009 libapriconv-1.dll
-rwxr-xr-x 1 ftp ftp          49234 Sep 28  2009 rotatelogs.exe
-rwxr-xr-x 1 ftp ftp          20564 Sep 28  2009 wintty.exe
-rw-r--r-- 1 ftp ftp          73782 Jan 18  2008 zlib1.dll

As you can see you don't get the filetime of files older than this month.

The MLSD command is much more accurate:

Code: Select all

Listing:	size=0;type=cdir;modify=20090722152124; .
Listing:	size=0;type=pdir;modify=20090722152124; ..
Listing:	size=1090546488;type=file;modify=20090426154910; adsl TV 2009-04-24 23-55-15 Oscar peterson   Studio Work [Mezzo].avi
Listing:	size=1734079107;type=file;modify=20090425011506; adsl TV 2009-04-24 23-55-15 Oscar peterson   Studio Work [Mezzo].mpg
Listing:	size=1365852470;type=file;modify=20090720092518; adsl TV 2009-07-20 08-25-10 C est pas sorcier S [France 3].mpg
Listing:	size=1373505504;type=file;modify=20090721092530; adsl TV 2009-07-21 08-25-18 C est pas sorcier S [France 3].mpg
Listing:	size=1365694195;type=file;modify=20090722092510; adsl TV 2009-07-22 08-25-10 C est pas sorcier S [France 3].mpg
Listing:	size=0;type=dir;modify=20090207191108; agk_tmp
Listing:	size=1282;type=file;modify=20081231192134; free16-9.xml
Listing:	size=1231;type=file;modify=20081231202910; free4-3.xml
Listing:	size=786432;type=file;modify=20090320095828; LA GRAINE ET LE MULET FR.DVDRIP.avi
Listing:	size=1291;type=file;modify=20090207174420; mezzo.xml

Combined with SITE UTIME respectively MFMT this would make the FTP-syncronisation very much more powerful for remote backups!

[ghisler(Author)]

I have made some tests with MLSD, but it lacks one extremely important information which is available with LIST: Unix links and their link targets!

[tumasch]

Yes, I saw that. Maybe it would be possible to enhance the LIST parser like that:

1. LIST
2. Parse list (like it's done now)
3. If MLSD-Flag ist set (managed by connection) send MLSD
4. Add the more accurate data (like modification date) to the parsed LIST data.

That's only a nice-to-have feature, but the improvement would be considerable.

Thanks for your work!

[ghisler(Author)]

That could indeed be done, but it would make directory changing twice as slow...

[cbdreinulldrei]

I understand the time this will eat up, but for the sake of synchronization of folders and files this would be wonderful to have.

I happen to have this problem with a FileZilla Server installation, which is just emulating Unix. I do not have any links in my FTP file system, so these info could be omitted.

Maybe Total Commander could offer this functionality and give the user a switch to chose from. Use MSLD and lose translation of links (if present) or use combined MSLD and LIST and lose some performance when syncing directories.

Already thought abount fiddling with FileZilla source but due to the author's opinions abot RFCs and workarounds, they would never make it into the official codebase, so would require changing every new version.

Having this functionality in TotalCommander would simply be great...

[tumasch]

yes, it slows the listing down. But there two possibilities to implement this:

- Making it optional in the syncronisation options

- Making it optional in the connection options

- use a second connection to speedup it (maybe to hard to implement ..)

Especially because the biggest advantage is to be able to syncronize more effectively using the accurate timestamps it could be usefull to use MSLD only in the sync process.

On the other hand, i'd like to have the correct filedates in the normal folder view too. Using a normal fast connection, I think nobody will notice a slow-down.

But I think it make sense to use it only optionally. But because not to much servers really offer this possibility, it could can be on by default.

[cbdreinulldrei]

Right. This feature would just help us over here a lot, too.

I have the option to use Microsoft's built-in FTP, which is a hassle to administrate, or use e.g. FileZilla Server, which is ok for what I want to do, but does present directories UNIX style.

So yes, having this as an option, and another option to combine it with LIST, this would be very helpful and much appreciated.

I use CyberDuck on a second machine here (running MacOS X), it uses MSLD and works just fine.

[cbdreinulldrei]

Sorry to ask this again, but it'd really help getting files synchronized if TC would make use of MSLD. Otherwise, all older files have a creation time of 0:00...

[ghisler(Author)]

I'm currently adding MLSD, and it works quite well with Unix links! Although I cannot see the link target, links are shown with
type=OS.unix=slink:

with Pure-FTPd! Therefore there will be no need to send both MLSD and LIST.

I wonder how this is handled with other servers, can anyone test it? Just write the following command on the command line during a connection:

MLST linkname

and post the result here. Please also write what ftp server you use. No need to post if you use Pure-FTPd too and it returns the same information.

[tumasch]

Unluckily I always get 503 Bad sequence of commands while sending the command manually. So I used Filezilla Client to connect and Wireshark to get the real communication:

Filezilla server (On windows server)

FEAT: (...) MLSD (...)

Code: Select all

type=dir;modify=20100113190043; data
type=dir;modify=20100103011953; Eigenes
type=file;modify=20100114021814;size=3; lw.z
type=file;modify=20100116124453;size=13638; mapdrive.bat
type=dir;modify=20100113075104; MSOCache
type=dir;modify=20100113190043; Programme
type=dir;modify=20100105140819; RECYCLER
type=dir;modify=20100113190043; Server
type=dir;modify=20100102135936; System Volume Information
type=dir;modify=20100113075055; _nonetz

Jana Server

FEAT: Doesn't support MLSD

Interesting, it uses MDTM to get file's exact modification date (to calculate UTC offset).

Code: Select all

Befehl:	LIST
Antwort:	150 File status okay; about to open data connection.
Antwort:	226 Closing data connection.
Status:	Berechne Zeitzonenabweichung des Servers...
Befehl:	MDTM lw.z
Antwort:	213 20100114021814
Status:	Zeitzonenabweichungen: Server: 3600 Sekunden. Lokal: 3600 Sekunden. Differenz: 0 Sekunden.
Status:	Anzeigen des Verzeichnisinhalts abgeschlossen

Pure-FTPd

FEAT: (...) MLSD (...)

MLSD:

Code: Select all

type=cdir;sizd=4096;modify=20090314124230;UNIX.mode=0755;UNIX.uid=19007;UNIX.gid=19007;unique=500ffe3g13d127; .
type=pdir;sizd=4096;modify=20091214125632;UNIX.mode=0710;UNIX.uid=19007;UNIX.gid=1301;unique=500ffe3gae048; ..
type=dir;sizd=4096;modify=20090314154301;UNIX.mode=0755;UNIX.uid=19007;UNIX.gid=19007;unique=500ffe3g13d132; intern
type=dir;sizd=4096;modify=20091222135224;UNIX.mode=0755;UNIX.uid=19007;UNIX.gid=19007;unique=500ffe3gaf218; public_html

LIST:

Code: Select all

drwxr-xr-x    3 sagognc  sagognc      4096 Mar 14  2009 .
drwx--x---   11 sagognc  1301         4096 Dec 14 13:56 ..
drwxr-xr-x   11 sagognc  sagognc      4096 Mar 14  2009 intern
lrwxrwxrwx    1 sagognc  sagognc        14 Jun 24  2009 public_html -> ../public_html

As you can see, you cannot identify filesystem links in MLSD result

Unknown Server software

Doesn't report MLSD in FEATure list, still works ??? Since it doesn't say what server software is running, there's no need to implement compatibility for this.

Code: Select all

modify=20030401152815;perm=flcdmpe;type=dir;unique=9U521EF9;UNIX.group=100;UNIX.mode=0705;UNIX.owner=14457; .
modify=20030401152815;perm=flcdmpe;type=dir;unique=9U521EF9;UNIX.group=100;UNIX.mode=0705;UNIX.owner=14457; ..
modify=20020921150520;perm=flcdmpe;type=dir;unique=9U521F02;UNIX.group=100;UNIX.mode=0705;UNIX.owner=14457; public_html

[ghisler(Author)]

Unluckily I always get 503 Bad sequence of commands while sending the command manually.

You cannot use MLSD manually, you need to use MLST ! Why? MLSD sends the list over a separate data connection, while MLST sends the result for a single file over the control connection.

Thanks for all the tests! You must have an old version of Pure-FTPd because it handles links fine here. I will give the user the choice between MLSD and LIST command in case he has such an outdated server.

Doesn't report MLSD in FEATure list, still works ?

Does it report MLST? The MLST feature indicates that both MLST and MLSD are supported. If none is reported, it would be a clear violation of RFC 3659:

http://tools.ietf.org/html/rfc3659#section-7.8

When responding to the FEAT command, a server-FTP process that supports MLST, and MLSD, plus internationalization of pathnames, MUST indicate that this support exists.

Anyway, TC will allow to enable MLSD even when the server doesn't indicate that it's there.

[tumasch]

Sorry, the server says

Code: Select all

FEAT
211-Features:
 MDTM
 MFMT
 MFF modify;UNIX.group;UNIX.mode;
 MLST modify*;perm*;size*;type*;unique*;UNIX.group*;UNIX.mode*;UNIX.owner*;
 REST STREAM
 SIZE
211 End

so it supports MLST and like you wrote, MLSD too. I just checked, it really works.


And concerning Pure-ftpd: I don't know which version is used, sice the server doesn't tell it:

Code: Select all

220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
220-You are user number 2 of 50 allowed.
220-Local time is now 12:59. Server port: 21.
220-This is a private system - No anonymous login
220 You will be disconnected after 15 minutes of inactivity.

It's a server at hostpoint.ch.

[ghisler(Author)]

Indeed according to RFC 3659, it's sufficient to show that MLST is supported, because the two are always supported together.

[Modem-Man]

I'm currently adding MLSD, and it works quite well with Unix links! Although I cannot see the link target, links are shown with
type=OS.unix=slink:
with Pure-FTPd! Therefore there will be no need to send both MLSD and LIST.

The symlink-to-MLST-implementation of a lot linux FTP servers is broken. See also "Errata ID: 1500" at <http://www.rfc-editor.org/errata_search.php?rfc=3659>

I wonder how this is handled with other servers, can anyone test it? Just write the following command on the command line during a connection:

MLST linkname

and post the result here. Please also write what ftp server you use. No need to post if you use Pure-FTPd too and it returns the same information.

I finished my HwFtpD Server testing (from Höft & Wessel Skeye AG). This works pretty well with TC7.56 and the MLST Implementation! But HwFtpD is only running on Skeye WinCE devices, so the Symlink problem can't never occur there.