New beta plugin: WebDAV (WebFolders)

[ghisler(Author)]

No, http/https is a very simple linear upload protocol, it should not be slower than FTPS.

[karnin]

Hmmmm, I am sorry, but FTPS has twice speed compared to https...

I did upgrade to OwnCloud 9.02: Same results with same upload to same server:
FTPS = 1100 Kb (Max for VDSL upload, ok!)
WebDAV/HTTPS = 550 Kb only

Win7-64pro/OpenSSL-1.02h/noProxy to HostEurope Apache with PHP-5.6 enabled, VDSL-50 connection 10ul/50dl.
No errors in log, no further problems, just the HTTPS upload speed is missing... (Download is ok, 5,5 Mb/s)

Any futher hints about this behaviour?

Thx + regards.

[Dalai]

Any futher hints about this behaviour?

Try other WebDAV clients (e.g. your browser) to see if it's a general problem. It might be possible to tweak the Apache webserver somehow (not sure).

Regards
Dalai

[karnin]

Did try another client (commercial SRT WebDrive Trial, https://southrivertech.com/products/webdrive/ ):
Upload speed is as expected about 900- 1000 Kb...
Screenshot: http://i.imgur.com/nMa6yQy.png

Tested with:
- OwnCloud-9.02/Apache/https
- https://webdav.magentacloud.de (Telekom)

TC-WebDAV2.8-plugin uploads with about 550 Kb to both servers.
So is seems to be an issue with WebDAV plugin, isn't it?

Best regards, k.

EDIT:
Did just crosscheck it with a Win-8.1-64 machine (TC+WebDAV28):
MagentaCloud + OwnCloud are uploading with 1000-1100 Kb, same internal network, same router, same firewall, same WebDAV target servers...

So: There are issues on Win7-64 machine possible? Probably SRT WebDrive uses own stack, so upload performance is much better. How to evaluate?

[Dalai]

So: There are issues on Win7-64 machine possible? Probably SRT WebDrive uses own stack, so upload performance is much better. How to evaluate?

Did you try to (temporarily) disable your anti-virus software or internet security suite? That might be scanning the HTTP traffic, but not FTP.

Regards
Dalai

[karnin]

Yes, security software had been disabled for further tests, no changes in upload rate...

[karnin]

Additional hint after removing Bitdefender BandWidthFix:

Upload to Margenta-Cloud or OwnCloud results in 'jumping' transfer rates between 735 kB and 1466 kB, pls refer to attached screenshot:
Image: http://i.imgur.com/ejGEWRj.png
(It shows two screenshots from same upload)

735 kB + 1466 kB = 2201 kB / 2 = 1100 kB

1100 kB is nearly the expected upload rate for VDSL 50 (10 Mbit upload)...
But why 'jumping'?

*confused*

Regards, k.

[ghisler(Author)]

Could be some kind of caching going on in the background...

[artt]

Setting a files modified time in a dav server for backup or synchronization software
Files that are uploaded to WebDAV servers typically get time stamped with the current date/time that the file was uploaded to the server, and not what the original source file date/time was, because WebDAV protocol does not support setting a files modified time or creation time. This make problems for file backup or synchronization software that relies on time stamps to determine if a file has been modified or not.

Workaround:
Many dav servers like "Yandex" and "pCloud" allow clients to set custom DAV properties, some like "Koofr", "PowerFolder" not, and Apache servers store custom properties (when enabled) in a .DAV folder under the folder in which the files are uploaded to.
So a dav client (like TC WebDav pluging) can set the following custom DAV properties on the server for the file in question:

Code: Select all

"srt_modifiedtime"  -  set to the modified time requested by the file system
"srt_creationtime" - set to the creation time requested by the file system
"srt_proptimestamp" - this value is set to whatever value the server returns for the "getlastmodified" property for this file

So when client uploads a local file called John.txt with a local date/time of "Dec 1, 1998 4:00PM" to the WebDAV server, it can set a property called "srt_modifiedtime" to "Dec 1, 1998 4:00 PM". When dav client requests the directory listing the server will respond with something that looks like this:

Code: Select all

PROPFIND /mydir
<filename> John.txt
<getlastmodifiedtime> March 22, 1:37 PM     - controlled by server, not dav client
<srt_modifiedtime> Dec 1, 1998 4:00 PM
<srt_proptimestamp> March 22, 1:37 PM

The server assigns "getlastmodifiedtime" to the current date/time, in this example March 22, 1:37PM.
When dav client parses this listing it sees the getlastmodifiedtime from the server, which is totally controlled by the server and can't be set by dav client.

Dav client also sees the "srt_modifiedtime" property and will use that time when displaying file times in the windows file system.
So the user will see "Dec 1, 1998 4:00 PM" as the files modified time even though it's not the file's actual time on the server.
This is useful because backup software will not see the intended time of the file.

There is one catch, and that is the other property called "srt_proptimestamp".
This is set by dav client to the value of "getlastmodifiedtime" at the time it sets the custom srt properties. This is done in case somebody else logs into the DAV server via RDP or SMB and changes the actual file date/time on the server file.
If dav client notices a difference between the srt_proptimestamp and the "getlastmodified" property by more than a few seconds, dav client will NOT use the srt_modifiedtime stamp and instead use the "getlastmodified" time.

It would be very useful if this supported by TC WebDav pluging.

Thanks

[ghisler(Author)]

The problem is that there is no standard, and no easy way for the user or the plugin to find out whether there are such custom fields. Any ideas?

[artt]

This is a working example with AutoHotkey for pCloud free storage.
(setting custom properties also working great with Yandex free storage - only the upload step is not working! )
SCRIPT STEPS
1. Propfind (FILE NOT EXIST ON SERVER YET)
2. Upload (FILE UPLOADED TO THE SERVER)
3. Proppatch (SET CUSTOM PROPERTIES TO FILE UPLOADED)
4. Propfind (FILE PROPERTIES INCLUDING OUR CUSTOM PROPERTIES)

Code: Select all

#NoEnv				; Prevents automatic resolution of empty variables to environment variables.
SetBatchLines, -1  		; Make the operation run at maximum speed.
#SingleInstance, force		; Skips the dialog box and REPLACES the old instance automatically (similar in effect to the Reload command).

Name_2 = pCloud
Burl_2 = https://webdav.pcloud.com
User_2 = <set your username>
Pass_2 = <set your password>

FilePath_Server_1 = /!!MFT__________/Torrent.ico
FilePath_Server_2 = /!!MFT__________/Torrent.ico
FilePath_Local_U1 = %A_ScriptDir%\!!MFT__________\Torrent.ico
FilePath_Local_U2 = %A_ScriptDir%\
FilePath_Local_D1 = %A_ScriptDir%\!!MFT__________\Torrent_Downloaded.ico
FilePath_Local_D2 = %A_ScriptDir%\

DAV_PROPFIND(Burl_2 FilePath_Server_1, User_2, Pass_2)
DAV_Upload(FilePath_Local_U1, Burl_2 FilePath_Server_1, User_2, Pass_2)
DAV_PROPFIND(Burl_2 FilePath_Server_1, User_2, Pass_2)
DAV_SetDateTime(FilePath_Local_U1, Burl_2 FilePath_Server_1, User_2, Pass_2)
DAV_PROPFIND(Burl_2 FilePath_Server_1, User_2, Pass_2)
Return



;================================================================================
DAV_Upload(FilePath, ServerPath, User, Pass) {
	;INIT
	_s:=chr(059)	; {;}
	_c:=chr(034)	; {"}
	whr := ComObjCreate("WinHttp.WinHttpRequest.5.1")
	whr.Open("PUT", ServerPath, True)
	
	;OPTIONS
	whr.SetRequestHeader("Content-Language", "content=" _c "en" _c)
	whr.SetRequestHeader("Accept-Language", "en")
	whr.SetCredentials(User,Pass,0)
	whr.SetRequestHeader("Depth", 0)
	whr.SetRequestHeader("User-Agent", "WebDav AHK")
	whr.SetRequestHeader("Referer", ServerPath)
	whr.SetRequestHeader("Translate", "f")
	;whr.SetRequestHeader("Content-Type", "text/xml`; charset=UTF-8")

	;UPLOAD
	ADODBObj := ComObjCreate("ADODB.Stream")	;Initialize the ADODB object to load binary files into the RAM
	ADODBObj.Type := 1				;Binary mode (adTypeBinary)
	ADODBObj.Open()					;Open the Stream object to load a binary file into it
	ADODBObj.LoadFromFile(FilePath)			;Load the contents of the binary file into the Stream
	whr.Send(ADODBObj.Read())			;Send the request with the binary code of our file in the request body
	ADODBObj.Close()				;Close the Stream object since we are done loading a file into it
	
	;RESPONSE
	whr.WaitForResponse(90)
	Status  := whr.Status()
	Headers	:= whr.GetAllResponseHeaders
	Text    := whr.ResponseText
	MsgBox,4096,%A_ThisFunc%,Status: [%Status%]`n`nHeaders:`n[%Headers%]`n`nText:`n[%Text%]
}

;================================================================================
DAV_SetDateTime(FilePath, ServerPath, User, Pass) {
	;INIT
	_s:=chr(059)	; {;}
	_c:=chr(034)	; {"}
	whr := ComObjCreate("WinHttp.WinHttpRequest.5.1")
	whr.Open("PROPPATCH", ServerPath, True)
	
	;OPTIONS
	whr.SetRequestHeader("Content-Language", "content=" _c "en" _c)
	whr.SetRequestHeader("Accept-Language", "en")
	whr.SetCredentials(User,Pass,0)
	whr.SetRequestHeader("Depth", 1)
	whr.SetRequestHeader("User-Agent", "WebDav AHK")
	whr.SetRequestHeader("Referer", ServerPath)
	whr.SetRequestHeader("Translate", "f")
	whr.SetRequestHeader("Content-Type", "text/xml`; charset=UTF-8")
	
	;CALCULATE CUSTOM TIMES
	xml := DAV_PROPFIND(ServerPath, User, Pass, 0)
	pr  := F_StringMidA(xml, ":creationdate>", "<")
	FileGetTime, mo, %FilePath%, M
	mo  := mo + A_NowUTC - A_Now
	FormatTime, mo, %mo%, yyyy-MM-dd'T'HH:mm:ss'Z'
	FileGetTime, cr, %FilePath%, C
	cr  := cr + A_NowUTC - A_Now
	FormatTime, cr, %cr%, yyyy-MM-dd'T'HH:mm:ss'Z'
	data=
	(
	<?xml version="1.0" encoding="utf-8" ?>
	<propertyupdate xmlns="DAV:" xmlns:u="mynamespace">
	   <set><prop>
	      <u:srt_proptimestamp>%pr%</u:srt_proptimestamp>
	      <u:srt_creationtime>%cr%</u:srt_creationtime>
	      <u:srt_modifiedtime>%mo%</u:srt_modifiedtime>
	   </prop></set>
	</propertyupdate>
	)

	;SEND - RESPONSE
	whr.Send(data)
	whr.WaitForResponse(90)
	Status  := whr.Status()
	Headers	:= whr.GetAllResponseHeaders
	Text    := whr.ResponseText
	MsgBox,4096,%A_ThisFunc%,Status: [%Status%]`n`nHeaders:`n[%Headers%]`n`nText:`n[%Text%]
}

;================================================================================
DAV_PROPFIND(url, User, Pass, ShowResponce=1) {
	;INIT
	_s:=chr(059)	; {;}
	_c:=chr(034)	; {"}
	whr := ComObjCreate("WinHttp.WinHttpRequest.5.1")
	whr.Open("PROPFIND", url, False)
	
	;OPTIONS
	whr.SetRequestHeader("Content-Language", "content=" _c "en" _c)
	whr.SetRequestHeader("Accept-Language", "en")
	whr.SetCredentials(User,Pass,0)
	whr.SetRequestHeader("Depth", 1)
	whr.SetRequestHeader("User-Agent", "WebDav AHK")
	whr.SetRequestHeader("Referer", url)
	whr.SetRequestHeader("Translate", "f")
	whr.SetRequestHeader("Content-Type", "text/xml`; charset=UTF-8")

	;REQUEST - RESPONSE
	;NOTE: Some services request body of PROPFIND to indicate specific properties
	;      (like those of other namespaces), in order to include them in response!
	data=
	(
	<?xml version="1.0" encoding="utf-8" ?>
	<propfind xmlns="DAV:" xmlns:u="mynamespace">
	   <prop>
	      <creationdate/>
	      <u:srt_proptimestamp/>
	      <u:srt_creationtime/>
	      <u:srt_modifiedtime/>
	   </prop>
	</propfind>
	)
	whr.Send(data)
	Status  := whr.Status()
	Headers	:= whr.GetAllResponseHeaders
	Text    := whr.ResponseText
	IF ShowResponce
		MsgBox,4096,%A_ThisFunc%,Status: [%Status%]`n`nHeaders:`n[%Headers%]`n`nText:`n[%Text%]
	Return Text
}

;================================================================================
F_StringMidA(Str, BeforePattern, AfterPattern="", Mode=1) { ;        v01-11-2015      Any Pattern NotFound ==> ""
 ; Mode=1: AfterPattern="" ==> All after BeforePattern               BeforePattern="" ==> All before AfterPattern
 ; Mode=2: AfterPattern="" ==> All after the last BeforePattern      BeforePattern="" ==> ""
 RETURN (Mode=1?((b1:=InStr(Str,(BP:=BeforePattern)))=0||(a:=InStr(Str,(AP:=AfterPattern),"",(b:=b1+StrLen(BP))))
 =0||(BP=""&&AP="")?"":SubStr(Str,b,(AP ?(a-b):StrLen(str)))):((b:=InStr(Str,(BP:=BeforePattern),"",-(StrLen(str)
 -(a:=(AP:=AfterPattern)=""?StrLen(str)+1:InStr(Str,AP))+1)))=0||BP=""?"":SubStr(Str,(c:=b+StrLen(BP)),a-c)))
} ; Mode=1: The 1st AP after the 1st BP                  //                  Mode=2: the 1st BP before the 1st AP

What the server is responding for each step above:

Code: Select all

---------------------------
DAV_PROPFIND (FILE NOT EXIST ON SERVER YET)
---------------------------
Status: [404]
Headers:
	[Connection: Keep-Alive
	Date: Tue, 08 Nov 2016 10:49:44 GMT
	Keep-Alive: timeout=5, max=99
	Content-Length: 309
	Content-Type: text/html; charset=iso-8859-1
	Server: Apache/2.4.10 (Debian)
	]
Text:
	[<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
	<html><head>
	<title>404 Not Found</title>
	</head><body>
	<h1>Not Found</h1>
	<p>The requested URL /!!MFT__________/Torrent.ico was not found on this server.</p>
	<hr>
	<address>Apache/2.4.10 (Debian) Server at webdav.pcloud.com Port 443</address>
	</body></html>
	]





---------------------------
DAV_Upload (FILE UPLOADED TO THE SERVER)
---------------------------
Status: [201]
Headers:
	[Connection: Keep-Alive
	Date: Tue, 08 Nov 2016 10:51:24 GMT
	Keep-Alive: timeout=5, max=99
	Content-Length: 71
	Content-Type: text/html; charset=ISO-8859-1
	Location: https://webdav.pcloud.com/!!MFT__________/Torrent.ico
	Server: Apache/2.4.10 (Debian)
	]
Text:
	[<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
	<html><head>
	<title>]

---------------------------
DAV_PROPFIND (PROPERTIES OF THE FILE ON SERVER - NO CUSTOM PROPERTIES YET)
---------------------------
Status: [207]
Headers:
	[Connection: Keep-Alive
	Date: Tue, 08 Nov 2016 10:52:56 GMT
	Keep-Alive: timeout=5, max=99
	Content-Length: 583
	Content-Type: text/xml; charset="utf-8"
	Server: Apache/2.4.10 (Debian)
	]
Text:
	[<?xml version="1.0" encoding="utf-8"?>
	<D:multistatus xmlns:D="DAV:" xmlns:ns1="mynamespace" xmlns:ns0="DAV:">
	<D:response xmlns:lp1="DAV:" xmlns:lp2="http://apache.org/dav/props/" xmlns:g0="mynamespace">
	<D:href>/!!MFT__________/Torrent.ico</D:href>
	<D:propstat>
	<D:prop>
	<lp1:creationdate>2016-11-08T10:51:24Z</lp1:creationdate>
	</D:prop>
	<D:status>HTTP/1.1 200 OK</D:status>
	</D:propstat>
	<D:propstat>
	<D:prop>
	<g0:srt_proptimestamp/>
	<g0:srt_creationtime/>
	<g0:srt_modifiedtime/>
	</D:prop>
	<D:status>HTTP/1.1 404 Not Found</D:status>
	</D:propstat>
	</D:response>
	</D:multistatus>
	]





---------------------------
DAV_SetDateTime (CUSTOM PROPERTIES SETTED)
---------------------------
Status: [207]
Headers:
	[Connection: Keep-Alive
	Date: Tue, 08 Nov 2016 10:56:00 GMT
	Keep-Alive: timeout=5, max=99
	Content-Length: 360
	Content-Type: text/xml; charset="utf-8"
	Server: Apache/2.4.10 (Debian)
	]
Text:
	[<?xml version="1.0" encoding="utf-8"?>
	<D:multistatus xmlns:D="DAV:" xmlns:ns1="mynamespace" xmlns:ns0="DAV:">
	<D:response>
	<D:href>/!!MFT__________/Torrent.ico</D:href>
	<D:propstat>
	<D:prop>
	<ns1:srt_proptimestamp/>
	<ns1:srt_creationtime/>
	<ns1:srt_modifiedtime/>
	</D:prop>
	<D:status>HTTP/1.1 200 OK</D:status>
	</D:propstat>
	</D:response>
	</D:multistatus>
	]





---------------------------
DAV_PROPFIND (FILE PROPERTIES INCLUDING OUR CUSTOM PROPERTIES)
---------------------------
Status: [207]
Headers:
	[Connection: Keep-Alive
	Date: Tue, 08 Nov 2016 10:57:59 GMT
	Keep-Alive: timeout=5, max=99
	Content-Length: 641
	Content-Type: text/xml; charset="utf-8"
	Server: Apache/2.4.10 (Debian)
	]
Text:
	[<?xml version="1.0" encoding="utf-8"?>
	<D:multistatus xmlns:D="DAV:" xmlns:ns1="mynamespace" xmlns:ns0="DAV:">
	<D:response xmlns:lp1="DAV:" xmlns:lp2="http://apache.org/dav/props/" xmlns:ns0="DAV:" xmlns:ns1="mynamespace">
	<D:href>/!!MFT__________/Torrent.ico</D:href>
	<D:propstat>
	<D:prop>
	<lp1:creationdate>2016-11-08T10:51:24Z</lp1:creationdate>
	<ns1:srt_proptimestamp>2016-11-08T10:51:24Z</ns1:srt_proptimestamp>
	<ns1:srt_creationtime>2016-11-08T06:58:38Z</ns1:srt_creationtime>
	<ns1:srt_modifiedtime>2014-08-21T06:43:41Z</ns1:srt_modifiedtime>
	</D:prop>
	<D:status>HTTP/1.1 200 OK</D:status>
	</D:propstat>
	</D:response>
	</D:multistatus>
	]

Best Regards

[ghisler(Author)]

Well, that wasn't my question, I know how to send requests to WebDAV servers.

The question is what field names should be used. How do you know that the server will support own namespaces, in this case "mynamespace"?

[artt]

How do you know that the server will support own namespaces?

I think it is the same as for FTP/SFTP - You can not know that the server preserves timestamps if you don't test it first (excuse me if i am wrong).
For webdav, you have to test if the server supports own namespaces by trying to add them. If not, you will skip the custom namespaces function.

what field names should be used?

If the server supports own namespaces, you can use
ANY NAME for namespace (in this case "mynamespace"),
ANY NUMBER of field names (in this case 3) and
ANY NAME for field names (in this case "srt_proptimestamp", "srt_creationtime" and "srt_modifiedtime").

Notes from my tests:
Most services include the custom properties in their response just using <allprop>, but
Some services request body of PROPFIND to indicate specific properties (like those of custom namespaces) in order to include them in response (ex. Yandex), so in this case <allprop> will not include the custom properties.

I have see two different response formats (depends on server) of PROPFIND request for custom properties:
a) <ns1:srt_modifiedtime>2014-08-21T06:43:41Z</ns1:srt_modifiedtime> (as this of my example) and
b) <srt_modifiedtime xmlns="mynamespace">2014-08-21T06:43:41Z</srt_modifiedtime>

If you set namespace name="http://www.southrivertech.com/" and fields "srt_proptimestamp", "srt_creationtime" and "srt_modifiedtime"
then your custom properties will be also recognized by (compatible with) "WebDrive" client (https://southrivertech.com/products/webdrive/)!

---

I am successfully using this technique about one year now using AutoHotkey scripts and free webdav storage services, and recently using WebDrive client and TC folder sync feature.
If you decide to add this feature to your webdav plugin i believe it would be a great upgrade and very helpful to those (like me) who need to keep synchronized files between home, office, desktop pc, laptop pc, .... without the need for a 3rd party application - only using Total Commander!

Thanks

[ghisler(Author)]

Thanks for all the details! One more question: What happens when a server doesn't support user namespaces? Do you get an error and the upload fails, or is it just ignored? In the former case, I would have to make the feature optional.

[artt]

What happens when a server doesn't support user namespaces? Do you get an error and the upload fails, or is it just ignored?

The upload never fails because of the user namespaces.
Our try to set custom namespaces to a file comes after the upload process, and if the server doesn't support user namespaces it just ignored.


I am describing what i do:

1. I upload the file as usually (nothing has changed up to here).

2. I send a PROPFIND request to get the "creationdate" of the just uploaded file (this is needed because the responce headers and text of the upload process does not contain this info)

3a. I send a PROPPATCH request with body:

Code: Select all

	<?xml version="1.0" encoding="utf-8" ?>
	<propertyupdate xmlns="DAV:" xmlns:u="%MyNamespace%">
	   <set><prop>
	      <u:srt_proptimestamp>%pr%</u:srt_proptimestamp>
	      <u:srt_creationtime>%cr%</u:srt_creationtime>
	      <u:srt_modifiedtime>%mo%</u:srt_modifiedtime>
	   </prop></set>
	</propertyupdate>

where (Time Format is [yyyy-MM-dd'T'HH:mm:ss'Z'] for all date-time variables below):
%MyNamespace% variable is my chosen custom namespace,
%pr% variable is the "creationdate" i get from step 2 above (this is needed - see the last paragraph of my post 320402 above),
%cr% variable is the local file's creationtime, and
%mo% variable is the local file's modifiedtime

3b. The response to this PROPPATCH request is always with Status:207 and the Headers are typical, no matters if the server accepts or not the command - supports or no custom namespaces.
The responce text is different for each server:
- <d:status>HTTP/1.1 200 OK</d:status>: one for all the new properties or one for every new property (3), no matters if the server accepts or not the command - supports or no custom namespaces.
- <D:status>HTTP/1.1 403 Forbidden</D:status>: one for all the new properties or one for every new property (3), if the server does not support custom namespaces.

4. I send a PROPFIND request to confirm that server supports custom namespaces (because by the 3b step above is not clear if it supports or not)
If the server does not support custom namespaces, i save it in an .ini file and next times i skip the 'not supported' steps.
This PROPFIND request must have body:

Code: Select all

	<?xml version="1.0" encoding="utf-8" ?>
	<propfind xmlns="DAV:" xmlns:u="%MyNamespace%">
	   <prop>
	      <creationdate/>
	      <u:srt_proptimestamp/>
	      <u:srt_creationtime/>
	      <u:srt_modifiedtime/>
	   </prop>
	</propfind>

(i have explain why in my previous post)

Thanks