FTP waits forever at 100%

[CCRDude]

I've been using TC as a FTP client for at least a year, if not longer. I'm using 7.02a now, but this bug has existed in all 7.0 versions I've tried, and maybe (not sure) in the latest 6.5x version before the switch to 7.0 as well.

When I upload files to any of a dozen servers (which are a mix; not at all the same provider, and I tried both active and passive FTP), the upload sometimes stops at 100%, still showing the transfer dialog and idling there forever. I can press "Cancel" and it reacts by trying to cancel there, but I have to compare the list of files and check which ones weren't transferred, so that using FTP really gets bothering sometimes

The thing that is really confusing me is that I didn't change any FTP settings and it worked for an eternity before using the new versions, and that it happens only at 100%, so it doesn't seem to be a server problem (but then, having nearly all servers change their behaviour at the same time looks somewhat unrealistic as well).

And to not have my first forum post here sound like a complaint only, let me add that TC is really a great piece of software - ever since I found some customization for my personal taste here on these forums, I couldn't even imagine working without it

[CoolWater]

Hi and welcome on board,

this problem has been discussed several times... Please use the search function next time

Take a look at the following threads:
http://ghisler.ch/board/viewtopic.php?t=16911&highlight=ftp+upload+100
http://ghisler.ch/board/viewtopic.php?t=10209&highlight=ftp+upload+100
http://ghisler.ch/board/viewtopic.php?t=8528&highlight=ftp+upload+100

HTH,
CoolWater

[CCRDude]

Just for your information: I have a forum for our own software, and I know how annoying it is to see the same question again and again. So guess what - I did use the search function before posting. Not only because it is annoying for regular visitors to see repeated questions, but also because it is annoying for me as a user to regularly check back for answers if I could find a solution by just searching

But... all the topics you linked to are actually different situations, and don't help me at all, nor did I feel to re-open an issue that seems to have been different.

First topic: WatchUer didn't have the problem with 7.01, while I had them since early 7.0 betas. Then he says 7.02a had fixed the problem, which it also has not for me (and yes I've been reading the change notes on previous releases to see if something like this might have been addressed as well ).

Second topic: timeout cannot be the reason since it happens in the middle of a bunch of files. Also, this posts seems to promise a fix with TC7, while it only started with TC7 for me - seems to be something different to me then.

Third topic: is from 2005, and I don't remember version 7.0 was available in 2005 already (remember I mentioned this started only with TC7, so a bug in 2005 doesn't seem to be the same that only recently popped up).

So what where you actually trying to tell me? That TC 7.0, or even TC 7.02a fixes the problem? That's what your quoted topics refer to, but that's what I posted did actually introduce the problem for me.

Sorry if this sounds like a harsh reply, but over-RTFMing sometimes seems to be almost as annoying as not using the search function (thanks though for your effort of doing a search as well )

[CoolWater]

Hi,

thanks for your kind reply What I tried to point out is that in most cases the problem is not TC but the "environment". Mostly ghisler(author) has some good ideas for finding the cause for the bugs, i.e. Firewall might drop the control channel because of too long idle time (large upload) or the server does not react on the "upload finished" message (which might not be your case since the server settings didn't change, as you said).

To make sure the problem is not TC-related, you can try another/other FTP clients like FileZilla to see if the problem still occurs.

What ghisler always likes to see is the ftp log... So maybe you could post it.

All of this "tips" also could be found in the linked topics

Regards,
CoolWater

[ghisler(Author)]

I also recommend that you try some different ftp client.

[CCRDude]

Hmmm you're right, I should have mentioned other clients as well. Whenever TC failed, I continued using WinSCP and SFTP. Ok, SFTP is another protocol...
At the same time I'm up- and downloading about 100 other files to a bunch of servers through my own mirror sync software using standard FTP through. Similar file size, same servers. Using Nevronas Indy 9 components for FTP, no problem at all in all the months TC had the problem.

There's a hardware router (ZyXEL) with a built-in firewall, which had caused problems in the past indeed - switching to passive mode helped with that back then, but not now. Disabling that firewall also didn't help. I did rule out that firewall before reporting
The other one is the ugly Windows XP one (64 bit, fully updated). Since that is absolutely standard I guess, I didn't test it ( ). Now that I did, disabling the Windows Firewall actually allowed all uploads to pass, at least in half a dozen tests to reproduce!

Anyway, to make my report complete, here's the log at the moment it hangs with Windows Firewall enabled:

Get directory
TYPE A
200 Type set to A
PORT 192,168,xx,yy,17,129
200 PORT command successful
LIST
150 Opening ASCII mode data connection for file list
Download
Waiting for server...
226 Transfer complete
TYPE I
200 Type set to I
PORT 192,168,xx,yy,17,134
200 PORT command successful
STOR 12345678.exe
150 Opening BINARY mode data connection for 12345678.exe
Upload

At this point, it has reached 100% and just waits there forever.

[ghisler(Author)]

I see - it sounds like the server is never receiving the data at all...

[CCRDude]

Oh, the server always has received everything. I did download those files a few times and compared the md5 hash - it matches, so the file has been uploaded completely (can't be the browser cache either since it was always updated files that were never before online there).

[CCRDude]

Ok, update: still happens with Windows Firewall disabled, just not so often. Last lines of log:

PASV
227 Entering Passive Mode (87,106,8,215,253,137).
STOR falite-peheader-xp.png
150 Opening BINARY mode data connection for falite-peheader-xp.png
Upload
Cancel pressed!
SIZE falite-peheader-xp.png

Pressed cancel after it idled at 100% for minutes. Then had to kill the TC process as usual since even after Cancel, it doesn't really do anything (except "command in progress").

Any further details I could provide?

[ghisler(Author)]

Sorry, I really can't help you. The server or firewall is dropping the ftp control connection during the transfer, and without notifying the client (e.g. connection reset), something which should NEVER actually happen. Therefore the Windows socket waits forever for a reply from the server.

[roentgen]

TC has to wait too?

[CCRDude]

Well, if you don't want to look into this problem, just say so, I can understand that since I seem to be a quite rare case! But please don't try to shoo me away with statements that are wild guesses and not true

And I don't really understand your argumentation. Even if the "server or firewall" would be dropping the connection without notification, it would do so only on if Total Commander is the FTP client, since a simple self-written FTP client does connect to the same servers, transmitting the same files, many dozen times a week, without this problem.

Well, even if you don't want to help me, I decided to spend some more time to maybe able to help you - well, admitted, also to proof you that your "server doesn't notify client" was just an invention to shoo me away - and monitored two sessions with Wireshark.

This FTP transfer was correct:

Code: Select all

84 19.095963 <client> <server> FTP  Request: TYPE I
85 19.130666 <server> <client> FTP  Response: 200 Type set to I
86 19.083264 <client> <server> FTP  Request: PASV
87 19.161037 <server> <client> FTP  Response: 227 Entering Passive Mode (A,B,C,D,210,224).
88 19.140676 <client> <server> FTP  Request: STOR testfile1.zip
89 19.158027 <server> <client> FTP  Response: 150 Opening BINARY mode data connection for testfile1.zip
90 19.390447 <client> <server> TCP  2893 > ftp [ACK] Seq=260 Ack=1517 Win=65468 [TCP CHECKSUM INCORRECT] Len=0
91 77.676547 <server> <client> FTP  Response: 226 Transfer complete
92 77.679617 <client> <server> FTP  Request: SIZE testfile1.zip
93 77.697071 <server> <client> FTP  Response: 213 2362365
94 77.709969 <client> <server> FTP  Request: TYPE A
95 77.739444 <server> <client> FTP  Response: 200 Type set to A
96 77.742220 <client> <server> FTP  Request: PASV
97 77.758795 <server> <client> FTP  Response: 227 Entering Passive Mode (A,B,C,D,232,97).
98 77.816462 <client> <server> FTP  Request: LIST
99 77.839387 <server> <client> FTP  Response: 150 Opening ASCII mode data connection for file list
100 77.946411 <client> <server> TCP  2893 > ftp [ACK] Seq=308 Ack=1676 Win=65309 [TCP CHECKSUM INCORRECT] Len=0
101 78.005369 <server> <client> FTP  Response: 226 Transfer complete
102 78.147183 <client> <server> TCP  2893 > ftp [ACK] Seq=308 Ack=1699 Win=65286 [TCP CHECKSUM INCORRECT] Len=0

And here's one transfer that idled eternally at 100%:

Code: Select all

103 121.559090 <client> <server> FTP  Request: TYPE I
104 121.648217 <server> <client> FTP  Response: 200 Type set to I
105 121.601392 <client> <server> FTP  Request: PASV
106 121.684367 <server> <client> FTP  Response: 227 Entering Passive Mode (A,B,C,D,230,230).
107 121.694043 <client> <server> FTP  Request: STOR testfile2.exe
108 121.767501 <server> <client> FTP  Response: 150 Opening BINARY mode data connection for testfile2.exe
109 121.913376 <client> <server> TCP  2893 > ftp [ACK] Seq=346 Ack=1832 Win=65153 [TCP CHECKSUM INCORRECT] Len=0
110 241.014555 <server> <client> FTP  Response: 226 Transfer complete
111 240.963945 <client> <server> TCP  2893 > ftp [RST, ACK] Seq=346 Ack=1855 Win=0 Len=0

As far as I can see that, the client gets notified on both occassions, at least if I interpret lines 91 and 110 correctly, so the response you renounced is actually there.

Admitted, there seems to be a connection reset (line 111), but that's 1. a reset that gets signaled to the client (so you could actually react to it), and 2. it doesn't affect the transfer, just your following, separate SIZE operation.

Oh, and btw: even if TC would wait for some socket operation... after half an hour, don't you think it could finally time out? Since there hasn't been ANY further FTP packet in Wireshark since half an hour, nor has the TC display of 100% transfer gone away!

PS #2: the TCP checksum incorrect thing is a Wireshark only thing, not a real problem, in case you haven't read about it yet.

[ghisler(Author)]

The problem is: I have already spent hundreds of hours troubleshooting similar ftp problems, but I have not made any real progress. It's a realy mystery to me what is happening with some TCP/IP connections. Functions like send() or recv() sometimes hang without an explainable cause. But I can't just abort them after a certain timeout, because some servers really take a long time to respond.

So all I can do is to recommend to use a standalone ftp client if you have such problems. The developper(s) of standalone ftp clients put all their time into just one function, ftp. In Total Commander, ftp is just one of many functions (like a screwdriver in a Swiss Army Knife). Therefore it can never be as sophisticated as a standalone ftp client.

[CCRDude]

Thanks for that an honest explanation

(regarding the screwdriver... that's exactly why I use "ready" components, like Synapse or Indy, in my own applications - I can forward any problems to a big community Kudos to you if your ftp code is all self-written )

[jcasares]

Oh, you're not alone in this! I had this issue since long ago with quite different FTP server configurations. It only happens with Total Commander so I'm sure it's not a server issue.

A shame because when you base all your work around TC having to resort to another FTP client is a nuisance.