FTP waits forever at 100%

[ghisler(Author)]

I'm sorry but there is no solution to this problem. The server just doesn't respond (servers usually send a "transfer complete" when done), so TC doesn't know whether the file was transferred completely or not.

[jcasares]

Have you read this post?

Well, if you don't want to look into this problem, just say so, I can understand that since I seem to be a quite rare case! But please don't try to shoo me away with statements that are wild guesses and not true

And I don't really understand your argumentation. Even if the "server or firewall" would be dropping the connection without notification, it would do so only on if Total Commander is the FTP client, since a simple self-written FTP client does connect to the same servers, transmitting the same files, many dozen times a week, without this problem.

Well, even if you don't want to help me, I decided to spend some more time to maybe able to help you - well, admitted, also to proof you that your "server doesn't notify client" was just an invention to shoo me away - and monitored two sessions with Wireshark.

This FTP transfer was correct:

Code: Select all

84 19.095963 <client> <server> FTP  Request: TYPE I
85 19.130666 <server> <client> FTP  Response: 200 Type set to I
86 19.083264 <client> <server> FTP  Request: PASV
87 19.161037 <server> <client> FTP  Response: 227 Entering Passive Mode (A,B,C,D,210,224).
88 19.140676 <client> <server> FTP  Request: STOR testfile1.zip
89 19.158027 <server> <client> FTP  Response: 150 Opening BINARY mode data connection for testfile1.zip
90 19.390447 <client> <server> TCP  2893 > ftp [ACK] Seq=260 Ack=1517 Win=65468 [TCP CHECKSUM INCORRECT] Len=0
91 77.676547 <server> <client> FTP  Response: 226 Transfer complete
92 77.679617 <client> <server> FTP  Request: SIZE testfile1.zip
93 77.697071 <server> <client> FTP  Response: 213 2362365
94 77.709969 <client> <server> FTP  Request: TYPE A
95 77.739444 <server> <client> FTP  Response: 200 Type set to A
96 77.742220 <client> <server> FTP  Request: PASV
97 77.758795 <server> <client> FTP  Response: 227 Entering Passive Mode (A,B,C,D,232,97).
98 77.816462 <client> <server> FTP  Request: LIST
99 77.839387 <server> <client> FTP  Response: 150 Opening ASCII mode data connection for file list
100 77.946411 <client> <server> TCP  2893 > ftp [ACK] Seq=308 Ack=1676 Win=65309 [TCP CHECKSUM INCORRECT] Len=0
101 78.005369 <server> <client> FTP  Response: 226 Transfer complete
102 78.147183 <client> <server> TCP  2893 > ftp [ACK] Seq=308 Ack=1699 Win=65286 [TCP CHECKSUM INCORRECT] Len=0

And here's one transfer that idled eternally at 100%:

Code: Select all

103 121.559090 <client> <server> FTP  Request: TYPE I
104 121.648217 <server> <client> FTP  Response: 200 Type set to I
105 121.601392 <client> <server> FTP  Request: PASV
106 121.684367 <server> <client> FTP  Response: 227 Entering Passive Mode (A,B,C,D,230,230).
107 121.694043 <client> <server> FTP  Request: STOR testfile2.exe
108 121.767501 <server> <client> FTP  Response: 150 Opening BINARY mode data connection for testfile2.exe
109 121.913376 <client> <server> TCP  2893 > ftp [ACK] Seq=346 Ack=1832 Win=65153 [TCP CHECKSUM INCORRECT] Len=0
110 241.014555 <server> <client> FTP  Response: 226 Transfer complete
111 240.963945 <client> <server> TCP  2893 > ftp [RST, ACK] Seq=346 Ack=1855 Win=0 Len=0

As far as I can see that, the client gets notified on both occassions, at least if I interpret lines 91 and 110 correctly, so the response you renounced is actually there.

Admitted, there seems to be a connection reset (line 111), but that's 1. a reset that gets signaled to the client (so you could actually react to it), and 2. it doesn't affect the transfer, just your following, separate SIZE operation.

Oh, and btw: even if TC would wait for some socket operation... after half an hour, don't you think it could finally time out? Since there hasn't been ANY further FTP packet in Wireshark since half an hour, nor has the TC display of 100% transfer gone away!

PS #2: the TCP checksum incorrect thing is a Wireshark only thing, not a real problem, in case you haven't read about it yet.

[ghisler(Author)]

Both connections look 100% OK, I don't see why TC shouldn't continue also in the second case. I can only imagine that something is preventing the "Transfer complete" message to reach TC. Maybe a personal firewall or so?

[Tenex100]

I have experienced the same exact problem on two FTP sites, I can not say for sure when the problem started because the last time I needed to upload a file via ftp was about 13 months ago.
I don't know if the following will help you, but now the problem went from
100% failure to 100% working.

All I did was invoke edit on the FTP connect screen and check:
"Use passive mode for transfers (like a WWW browser"
and
"Send command to keep connection alive"
BTW the command is a NOOP

Regards

[ghisler(Author)]

TRY using LIST instead of NOOP. Most current ftp servers will ignore NOOP and drop the connection anyway.

[fredscal]

I must sadly confirm that, while the FTP feature is a plus to TC, it can never replace a real FTP client.

It is convenient for the occasional uploading of single files, to be able to do it from inside TC, but for the frequent uploading of large batches it is just not reliable enough.

[limex]

I definitely agree with fredscal.
I love TC for all the file handling it provides. It saves so much time time when you are coding php and deal with so many files.
It is so great that I am really upset when I have to work on a PC without TC. Therefore I always have my Portable TC with me.
It is the first tool I install when I setup a PC.

But:
If you want to sync your code files from your local development environment to a server and you have to be very careful ... copy updated file to the server, but don't overwrite config files of the server ...

To do that with TC is a major pain in the ass: No blacklist for files to exclude from sync, problems with reconnect when the connection times out, ....

FTP with TC is fine for a handful of files when you have the chance and the overview what to re-try in the case of an connection error.

It seems to me that TC is more vulnerable to server connection issues than other FTP progs. But most FTP progs don't come with left/right folder view and sync features.

I personally recommend Beyond Compare. It is robust and the cost is below €40.

[Zom-B]

I've had the same issue since about a year now. (didn't do much ftp in the past). The FTP server is irrelevant and the client ISP is (most probably) irrelevant. The problem happened to me both while in Holland and in Japan.

TC only hangs at 100% on big files. If I upload a random folder with small and big files, the small continue fine and after the big file it hangs. Since my upload speed is always constant regardless of server, it can be either of these problems:
-it happens when the file is too large (4.8MB is already large)
-it happens when the file takes a certain amount of time to reach 100% (75 seconds is already long)

Funny thing is, it doesn't hang when I upload a single file, only when I select two or more files.

Single file transfer:

Code: Select all

1:TYPE I
1:200 Type set to I
1:PASV
1:227 Entering Passive Mode (50,115,174,169,231,39).
1:STOR ch03.7z
1:150 Opening BINARY mode data connection for ch03.7z
1:Upload
1:Upload: 10,752 bytes
<snip>
1:Upload: 4,988,173 bytes, 64 kbytes/s
1:226 Transfer complete
1:SIZE ch03.7z
1:213 4988173
1:Copied (13/03/2013 1:58:41): j:\@Projects\@Orinjido\rawbackup\ch03.7z -> ftp://storage.orinjido.info/var/ostorage/raws/ch03.7z 4,988,173 bytes, 62 kbytes/s

Two file transfer:

Code: Select all

1:TYPE I
1:200 Type set to I
1:PASV
1:227 Entering Passive Mode (50,115,174,169,135,202).
1:STOR ch03.7z
1:150 Opening BINARY mode data connection for ch03.7z
1:Upload
1:Upload: 10,752 bytes
<snip>
1:Upload: 4,988,173 bytes, 69 kbytes/s, 2 m
1:Cancel pressed!
1:SIZE ch03.7z
1:OFFLINE2, error=10054
1:Copied (13/03/2013 2:05:54): j:\@Projects\@Orinjido\rawbackup\ch03.7z -> ftp://storage.orinjido.info/var/ostorage/raws/ch03.7z 4,988,173 bytes, 61 kbytes/s

[limex]

Hi,
great that someone adds input to this still unsolved issue.
In the meantime I do sync/upload tasks with beyond compare.
It is quick and stable.

[ghisler(Author)]

Windows error 10054 means:
An existing connection was forcibly closed by the remote host.

[Abmea]

Perhaps a bit too late for many of the contributors on this topic, but I believe I have found the reason for the problem.

The most likely cause for this problem is that the TCP connection on the FTP control channel (not the data channel) is closed or otherwise no longer operational. This can be either because of a NAPT or TCP timeout of because the server closes the connection after x seconds of inactivity (explaining the "An existing connection was forcibly closed by the remote host" error).

The reason the TCP connection is closed is because TC does NOT send the keep-alive user data on the control channel during a file transfer (these keepalive messages can be configured in the FTP settings).
If the transfer time then takes too long for the various time-outs, the control channel is dead by the time the transfer is done (and TC wants the use the control channel to send a SIZE command).

In my opinion, this also explains the many different facets and reported solutions to this problem.
The TCP connection can be closed (or remain open) due to a number of reasons. For example, many TC users will use some form of TCP proxy and/or VPN tunnel for their FTP connections. Often, proxies and VPN connections use TCP keepalive and thus hide the problem.

In my situation, I can reproduce/resolve the problem by manipulating settings like:
-FTP server control channel timeout.
-NAPT timeouts
-The use of TCP keepalive in the underlying network.

Still, the best solution would still be a bug-fix in TC.
If TC sends the user-data keepalive messages on the control channel during a transfer (like it would without the transfer) then this problem should be fixed.

[ghisler(Author)]

The reason the TCP connection is closed is because TC does NOT send the keep-alive user data on the control channel during a file transfer (these keepalive messages can be configured in the FTP settings).

There is no such thing for most ftp servers. I have made many tests, and it seems that sending new line characters works with many servers.

You can set the following option in wcx_ftp.ini under [General]
KeepAliveTransfer=1

Explanation:
1: Send keepalive also during a transfer. This will send just newline characters to the control connection to avoid that a firewall/gateway drops the connection. Set also individually per server (the latter not working in BTM). Note that this will confuse some FTP servers, but it may be the last resort in case of frequent connection losses.