I only adapted the changes in SftpDownloadFileW() to b4 and compiled it like before. Here, SFTP mode now performs even better than SCP mode this shouldn't be possible at all SFTP mode reaches about 33 MB/s while SCP mode only makes 22 MB/s.
I checked your changes - apparently you start with a read block size of 60000 bytes, and increase it up to 1.9 MBytes until the function fails, then you reduce it again. My code uses a fixed block size of 32kBytes.
I'm not surprised that it's even faster than SCP, because you use a fixed block size for SCP.
remittor wrote: ↑2020-11-05, 14:27 UTC
I can assume that it is only useful for people who use standard Notepad utility. But the standard Notepad already has support for LF EOL.
I don't know whether your assumption has legs, but lets roll with it. The CR/LF conversion was added to the plug-in around December 2008. And, well, Notepad didn't have LF support then. It didn't have it until the beginning of 2018. That's a butt-load of time right there between the end of 2008 and beginning of 2018. Also, this only holds true for Notepads on Win10. On any other Win version supported by TC, i guess the butt-load of time between the plug-in having introduced CR/LF conversions and Notepad supporting Unix-style LF approaches infinity.
According to the FTP specification, ASCII files are always transferred using a CR+LF pair as line ending.
So in case the file is transferred from the client to the server, the client has to make sure CR+LF is used. Therefore it has to add nothing (on Microsoft Windows),...
The sender converts the data from an internal character representation to the standard 8-bit NVT-ASCII representation (see the Telnet specification). The receiver will convert the data from the standard form to his own internal form.
In accordance with the NVT standard, the <CRLF> sequence should be used where necessary to denote the end of a line of text.
The SFTP-plugin should not perform any conversions when using text mode.
It turns out that the plugin is trying to emulate text mode
2remittor
SFTP has nothing to do with FTP or FTPS, despite the similarity in name. So neither of your quotes apply here. Besides, you can just disable the conversion if you don't like it.
Regards
Dalai
#101164 Personal licence
Ryzen 5 2600, 16 GiB RAM, ASUS Prime X370-A, Win7 x64
2remittor
No, it's not the same between FTP and SFTP. You will have to read the specs more carefully.
The FTP spec encourages line endings to be CR/LF, where the sender should make sure all line endings are CR/LF; that includes the client should convert where necessary when the client is the sender (It's in the quote you provided above.)
While the default canonical line break in SFTP is also CR/LF, the SFTP protocol allows for this canonical form to be changed by the server (a facility the FTP protocol does not provide). And unlike FTP, the SFTP spec discourages the client from doing line ending conversions when encountering a non-canonical line ending, regardless of the client being sender or receiver. (It's in the section of the SFTP spec you linked to.)
I guess it would be of benefit for discussions about the SFTP plug-in if we stop making comparisons with or interpretations of unrelated specs and protocols (regardless of them being inaccurate or not), as doing so will inevitably muddle the discussion.
(P.S: Also, my previous comment was about your Notepad assumption and about Notepad support for Unix-style line endings. Didn't say anything about SFTP or the SFTP plug-in there...)
ghisler(Author) wrote: ↑2020-11-03, 16:50 UTC
I checked your changes - apparently you start with a read block size of 60000 bytes, and increase it up to 1.9 MBytes until the function fails, then you reduce it again. My code uses a fixed block size of 32kBytes.
I'm not surprised that it's even faster than SCP, because you use a fixed block size for SCP.
Is there any disadvantage using dynamic block sizes?
It doesn't seem to check how long it takes to transfer one block. So when transferring files over a slow connection (e.g. via smartphone access point), it could take several seconds per block during which the operation couldn't be aborted.
Ok, that sounds reasonable. Nevertheless, it should be at least possible to set an INI switch to enable dynamic block size. Or maybe the logic could be improved, i.e. start with a smaller block size, check block transfer time and increase/decrease block size accordingly unless no error occurs.
Just leaving as is seems not to be the right way when much more speed is possible in theory...
CoolWater wrote: ↑2020-11-17, 16:30 UTC
Or maybe the logic could be improved...
This is quite feasible. Always make the Buffer size maximum (1.9 MB), and retrieve information about the received data from the internal structure of libssh2.
To do this, you need to include in the project the internal headers of libssh2.