TC hangs when downloading huge files over FTP

[foobar]

Hi all, I noticed an annoying bug in TC, which happens when downloading huge (4-10GB) files over ftp.

The download starts ok, with about 25MB/s transfer rate (vsftpd on my home LAN), but then roughly every 30 seconds, TC stops responding for about 10 seconds - the screen freezes, there is no network activity and windows task manager marks the program as not responding.
TC then recoveres and continues downloading, but hangs again every 30 seconds or so.
According to both the vsftpd log, and the TC ftp log, the connection was never dropped, so there has been no problem on the server side.

This is probably a problem in TC, as I do not experience this behavior when using the Filezilla FTP client.
Both ftp clients are set up the same way - passive mode, anonymous user, keepalive - changing any of these settings didn't have any effect.

I'm runing Windows 7 64-bit.

[Hacker]

Hello foobar,
An antivirus program perhaps? Or writing to disk from cache? Could you perhaps check with Process Monitor what process is active (CPU / HDD-wise) during those 10 seconds?

Roman

[foobar]

hi, during the 10 seconds, the system is basically idle - CPU load is around 5% (Core i7 860), HDD is idling as well with basically no writes (WD Caviar Black 640GB). The vsftpd server is idle as well.
Disabling my Nod32 antivirus did not have any effect on this.

These slowdowns are not present when I use Filezilla with the same settings and downloading to the same HDD.

[ghisler(Author)]

Can you post a log of the transfer? Maybe the connection is lost again and again, so TC has to reconnect and resume?

[foobar]

there is nothing relevant in the log, and I am not allowed to post it, as I just registered here today and there is some URL in the log

But it has probably something to do with the disk caching after all - when downloading to an external drive (connected with usb), TC hangs for about 15 seconds, when downloading to an internal drive, TC hangs for about 4 seconds. The interval is roughly the same - about every 30 seconds.

Is there any way how this could be optimized ?

[ghisler(Author)]

Downloading directly to USB isn't a good idea because writing is NOT cached by Windows. Downloading to internal drives gives no delay at all, on none of the tested computers, so something must be slowing down your system every 30 seconds. I recommend that you make a scan for malware (troyans, viruses etc.), if possible from a standalone boot CD.

[foobar]

the thing is, that there are no such slowdowns when using Filezilla to download the same files to the same external drive.

I'm quite confident my PC is clean, and I experience these slowdowns only when using TC to download large files from FTP .
In fact - even during these slowdowns, the PC is absolutely resposive and everything works as usual - it's just the TC window that fails to redraw and respond to any actions.

Since it has been some time since the first install of TC, I have also tried a clean install, removing everything except for my registration key, to no effect

any other ideas ?

..elaborating a bit more - I have a dual-boot windows 7 64bit and windows XP 32bit on the PC - this issue appears only while running the windows 7. In XP the downloads are ok.

I also tried again downloading the same 4GB file using Filezilla first, than TC (so TC probably had a bit advantage even from the vsftpd server cache).
But it took 158 seconds to download with Filezilla, and 200 seconds with TC because of the slowdowns.
The PC was idle all the time, the FTP server as well.

[ghisler(Author)]

I have analyzed the download code now: TC calls FlushFileBuffers every 30 seconds during a long download. This is done for several reasons:
1. It causes cached data to be written to disk immediately, so it isn't lost in case of a power loss or so
2. It updates the file size in the directory, so you can access the file already during the download (e.g. look at the already downloaded part of a huge zip archive)

However, I cannot see any large slowdown caused by this function. It takes about 2 seconds each time here (Windows 7 x64).

[foobar]

would it make sense to call this function in a separate thread and have the transfer asynchronous ?

I understand that transferring large files to a USB drive (no matter how fast) will have some performance drawbacks, however, the application should not stop responding because of this IMHO

Im not praising the other ftp app, its just a proof that it can be done.. ..and the source codes are available

[ghisler(Author)]

Unfortunately it's not that easy, the entire download would have to be done in the background (which is currently impossible, sorry). The other apps simply don't call "Flush", but don't allow to access the partial file during the download either.

[foobar]

how about a checkbox to disable flushing then ?

I mean there's really not much you can do with an incomplete file during the transfer - if you try to open an incomplete zip/rar/etc archive, you get an error message that the archive is corrupted ("Error in packed file" in case of TC). You can not play back an incomplete video file most of the time, in fact Im not sure if there are any file formats which would make sense to access while the file is incomplete. So concerning this point, I dont really see any added value compared to the non-flushing approach. Also, the file is still "unflushed" during the 30 seconds between the last, and the next flush

And in the event of a "catastrophic failure", Im sure an incomplete part of a file that can be downloaded from a FTP server would be the least of my concerns - possible loss of data from other running desktop applications and Windows components themselves, would be probably a lot more critical.
Also, I have not experienced these slowdowns while copying data between 2 drives on the same PC - so if copying 2 files on the same PC is safe enough - not to require flushing during the process - what makes downloading a file available on a FTP server not "safe enough" ?

I respect whatever decision you take, Im just trying to provide my insights

[ghisler(Author)]

I will consider it, thanks for your suggestion.