various FTP upload errors (upload size mismatch)

English support forum

Moderators: white, Hacker, petermad, Stefan2

zsero
Junior Member
Junior Member
Posts: 27
Joined: 2007-02-01, 15:03 UTC

various FTP upload errors (upload size mismatch)

Post by *zsero »

I have had strange FTP problems with my host using TC 7.50. I have just tried 7.55 but it's the same.

The problem is:
- 1 out of 10 files during upload gets an error
- when editing using F4 (download - edit - close - upload), it almost always gets an error.

The error in the FTP log is "upload size mismatch". Maybe my host is special, but
- only TC thinks it's an error, in reality the transfer works fine
- the same host with FlashFXP, Filezilla, etc. works fine

I tried every possible transfer mode, PASSIVE on, off, but no change.
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

TC sends a SIZE command just after the upload to verify that the file is OK. If the SIZE command reports an invalid size, an error will be shown. Can you check in the log what size it reports? Maybe the server performs some kind of data conversion even in binary mode?

If you are 100% sure that the files are complete, you can turn off the size check like this:

wcx_ftp.ini, section [General] , add the line:
SpecialFlags=32

See the help for what it means (section about wcx_ftp.ini).
Author of Total Commander
https://www.ghisler.com
zsero
Junior Member
Junior Member
Posts: 27
Joined: 2007-02-01, 15:03 UTC

Post by *zsero »

I will look in the specialflags soon. Here is a FTP log of a transfer which completed fine but said it had errors.

Code: Select all

----------
Connect to: (2010.04.23 15:18:32)
hostname=
username=
startdir=/website/www
=87.229.73.77
220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
220-You are user number 2 of 200 allowed.
220-Local time is now 15:18. Server port: 21.
220-This is a private system - No anonymous login
220-IPv6 connections are also welcome on this server.
220 You will be disconnected after 15 minutes of inactivity.
USER 
331 User  OK. Password required
PASS ***********
230-User xxx_hu has group access to:  3771    
230 OK. Current directory is /website/www
SYST
215 UNIX Type: L8
FEAT
211-Extensions supported:
 EPRT
 IDLE
 MDTM
 SIZE
 REST STREAM
 MLST type*;size*;sizd*;modify*;UNIX.mode*;UNIX.uid*;UNIX.gid*;unique*;
 MLSD
 TVFS
 ESTP
 PASV
 EPSV
 SPSV
 ESTA
 AUTH TLS
 PBSZ
 PROT
 UTF8
211 End.
CWD /website/www
250 OK. Current directory is /website/www
Connect ok!
PWD
257 "/website/www" is your current location
Get directory
TYPE A
200 TYPE is now ASCII
PASV
227 Entering Passive Mode (87,229,73,77,243,237)
LIST
150 Accepted data connection
Download
Waiting for server...
226-Options: -a -l 
226 104 matches total
CWD al
250 OK. Current directory is /website/www/al
PWD
257 "/website/www/al" is your current location
Get directory
PASV
227 Entering Passive Mode (87,229,73,77,244,73)
LIST
150 Accepted data connection
Download
Waiting for server...
226-Options: -a -l 
226 10 matches total
CWD images
250 OK. Current directory is /website/www/al/images
PWD
257 "/website/www/al/images" is your current location
Get directory
PASV
227 Entering Passive Mode (87,229,73,77,242,252)
LIST
150 Accepted data connection
Download
Waiting for server...
226-Options: -a -l 
226 5 matches total
TYPE I
200 TYPE is now 8-bit binary
PASV
227 Entering Passive Mode (87,229,73,77,245,222)
RETR fejlec.png
150-Accepted data connection
150 7.7 kbytes to download
Download
Waiting for server...
226-File successfully transferred
226 0.000 seconds (measured here), 159.94 Mbytes per second
Copied (2010.04.23 15:18:50): ftp://ftp7.yyy.hu/website/www/al/images/fejlec.png -> C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\1\_tc\fejlec.png 7 877 bytes, 248.0 kbytes/s
PASV
227 Entering Passive Mode (87,229,73,77,244,94)
STOR fejlec_projektoldal_3_kozter.jpg
150 Accepted data connection
Upload: 32 796 bytes, 137.5 kbytes/s
226-File successfully transferred
226 2.649 seconds (measured here), 12.09 Kbytes per second
SIZE fejlec_projektoldal_3_kozter.jpg
213 32796
Copied (2010.04.23 15:18:56): P:\web\Kepek_cimlap\fejlec_projektoldal_3_kozter.jpg -> ftp://ftp7.yyy.hu/website/www/al/images/fejlec_projektoldal_3_kozter.jpg 32 796 bytes, 11.7 kbytes/s
Get directory
TYPE A
200 TYPE is now ASCII
PASV
227 Entering Passive Mode (87,229,73,77,243,185)
LIST
150 Accepted data connection
Download
Waiting for server...
226-Options: -a -l 
226 6 matches total
CDUP
250 OK. Current directory is /website/www/al
PWD
257 "/website/www/al" is your current location
CWD projektek
250 OK. Current directory is /website/www/al/projektek
PWD
257 "/website/www/al/projektek" is your current location
Get directory
PASV
227 Entering Passive Mode (87,229,73,77,242,157)
LIST
150 Accepted data connection
Download
Waiting for server...
226-Options: -a -l 
226 3 matches total
CWD 3_kozter
250 OK. Current directory is /website/www/al/projektek/3_kozter
PWD
257 "/website/www/al/projektek/3_kozter" is your current location
Get directory
PASV
227 Entering Passive Mode (87,229,73,77,245,167)
LIST
150 Accepted data connection
Download
Waiting for server...
226-Options: -a -l 
226 9 matches total
CDUP
250 OK. Current directory is /website/www/al/projektek
PWD
257 "/website/www/al/projektek" is your current location
CDUP
250 OK. Current directory is /website/www/al
PWD
257 "/website/www/al" is your current location
SIZE index.php
213 1495
TYPE I
200 TYPE is now 8-bit binary
PASV
227 Entering Passive Mode (87,229,73,77,243,101)
RETR index.php
150 Accepted data connection
Download
Waiting for server...
226-File successfully transferred
226 0.000 seconds (measured here), 33.98 Mbytes per second
Copied (2010.04.23 15:19:04): ftp://ftp7.yyy.hu/website/www/al/index.php -> C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\1\_tc\index.php 1 495 bytes, 97.7 kbytes/s
PWD
257 "/website/www/al" is your current location
PASV
227 Entering Passive Mode (87,229,73,77,243,78)
STOR index.php
150 Accepted data connection
Upload: 1 517 bytes
226-File successfully transferred
226 0.021 seconds (measured here), 69.19 Kbytes per second
SIZE index.php
213 1495
Upload size mismatch!
Copied (2010.04.23 15:19:10): C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\1\_tc\index.php -> ftp://ftp7.yyy.hu/website/www/al/index.php 1 517 bytes, 31.3 kbytes/s
I think its the servers problem as the uploaded file should be 1517 bytes and it only reported this size after a F2 refresh. It seems that the server has some delay in refreshing the file size.

It seems that only the specialflag will be my solution!
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

OK, I think I see the problem: TC sets the transfer mode to Binary here:

Code: Select all

TYPE I
200 TYPE is now 8-bit binary
Later TC uploads the file in binary mode, but it seems that the server still performs a conversion from Windows to Unix line ends!

Code: Select all

PASV
227 Entering Passive Mode (87,229,73,77,243,78)
STOR index.php
150 Accepted data connection
Upload: 1 517 bytes
226-File successfully transferred
226 0.021 seconds (measured here), 69.19 Kbytes per second
SIZE index.php
213 1495
Upload size mismatch! 
As you can see, the resulting size is smaller!
Solution: Change the transfer mode from Binary to Automatic, and make sure that *.php is in the list. Total Commander then expects that the file size can change on uploads, something which should never happen when doing binary uploads.
Author of Total Commander
https://www.ghisler.com
zsero
Junior Member
Junior Member
Posts: 27
Joined: 2007-02-01, 15:03 UTC

Post by *zsero »

Solution: Change the transfer mode from Binary to Automatic, and make sure that *.php is in the list. Total Commander then expects that the file size can change on uploads, something which should never happen when doing binary uploads.
It doesn't help, the server remembers the original size before a re-upload, and always reports the file size what it was before the re-upload. I tried binary, text, auto with php, no change.

I need to push F2 manually to fix the file size value displayed after an upload. It seems that only the special fix would work (strange, since the recommended client for my host is TC). Can I apply the special fix on a FTP site basis or its only a global off, global on switch?
Henrie
Member
Member
Posts: 194
Joined: 2006-09-03, 23:12 UTC
Location: Volkel, the Netherlands

Post by *Henrie »

It doesn't help, the server remembers the original size before a re-upload, and always reports the file size what it was before the re-upload. I tried binary, text, auto with php, no change.
Solution: Change the transfer mode from Binary to Automatic, and make sure that *.php is in the list. Total Commander then expects that the file size can change on uploads, something which should never happen when doing binary uploads.
With this Ghisler meant that in your wcx_ftp.ini see if you have in the [General] section a value ASCII=*.php
You can open wcx_ftp.ini via Configuration > Change Settings File Directly.

My line for ASCII is:
ASCII=*.txt *.htm *.html *.php *.pl *.cgi *.js *.css *.inc .htaccess .htpasswd .htgroup .forward .procmailrc
zsero
Junior Member
Junior Member
Posts: 27
Joined: 2007-02-01, 15:03 UTC

Post by *zsero »

Henrie wrote:
It doesn't help, the server remembers the original size before a re-upload, and always reports the file size what it was before the re-upload. I tried binary, text, auto with php, no change.
Solution: Change the transfer mode from Binary to Automatic, and make sure that *.php is in the list. Total Commander then expects that the file size can change on uploads, something which should never happen when doing binary uploads.
With this Ghisler meant that in your wcx_ftp.ini see if you have in the [General] section a value ASCII=*.php
You can open wcx_ftp.ini via Configuration > Change Settings File Directly.

My line for ASCII is:
ASCII=*.txt *.htm *.html *.php *.pl *.cgi *.js *.css *.inc .htaccess .htpasswd .htgroup .forward .procmailrc
It doesn't work, not even in Text mode. As I have seen other client's doesn't even use SIZE, so I think the specialflag shouldn't be a big trouble.

Ghisler, if you would like, I can give you a test account for a week or something.
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

That wouldn't help me if the server returns invalid data after SIZE. I added the SIZE check because some servers lost data on uploads, the uploaded files were incomplete. That's why I will not remove the size check after upload. Your only hope is that the server maker will fix the bug in his server.
Author of Total Commander
https://www.ghisler.com
zsero
Junior Member
Junior Member
Posts: 27
Joined: 2007-02-01, 15:03 UTC

Post by *zsero »

Thank you, the SpecialFlags=32 works very well!

So now my problem is more of a feature request, which would be to implement SpecialFlags=32 on site level, not on global level. (Optionally a nice checkbox would be super on the site editor). Thanks for the help!
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

You can add the SpecialFlags=32 line to each connection manually in wcx_ftp.ini. The line in [General] will be used only when there is no server-specific line.
Author of Total Commander
https://www.ghisler.com
Fosfor
New Member
New Member
Posts: 1
Joined: 2011-05-03, 12:25 UTC
Location: Czech rep., Prague
Contact:

Post by *Fosfor »

Hi,

I had the same problem with webhosting cesky-hosting.cz. After short mail discussion they have admited that problem is at their server - antivirus program holds the uploaded files for the check and make it available after short while. And when TC check the filesize by SIZE command in this while, it gets the old one.

Solution (for me) is SpecialFlags=32, but it would be nice to have some configurable possibility to delay SIZE check. But don't know how it can be usable in high trafic :? (Batch check at the end of the transmission?)

Definitely thanks for help and hope this info can help someone else (this thread is the first for google "total commander upload size mismatch" 8) )

Fosfor
zsero
Junior Member
Junior Member
Posts: 27
Joined: 2007-02-01, 15:03 UTC

Post by *zsero »

Fosfor wrote:Hi,

I had the same problem with webhosting cesky-hosting.cz. After short mail discussion they have admited that problem is at their server - antivirus program holds the uploaded files for the check and make it available after short while. And when TC check the filesize by SIZE command in this while, it gets the old one.

Solution (for me) is SpecialFlags=32, but it would be nice to have some configurable possibility to delay SIZE check. But don't know how it can be usable in high trafic :? (Batch check at the end of the transmission?)

Definitely thanks for help and hope this info can help someone else (this thread is the first for google "total commander upload size mismatch" 8) )

Fosfor
For me I just really don't understand why there is still no option in the GUI for specifying SIZE check on a site-by-site basis. There are lot of sites which has some kind of strange server, where you need to bypass SIZE check. But how many users go as far as here and dig into this forum and edit the ftp.ini by hand just to bypass the SIZE check. For me it would be almost as important as any other option on the advanced page for the FTP sites.
Macbetht
New Member
New Member
Posts: 1
Joined: 2011-10-07, 12:51 UTC
Location: 121312er

Post by *Macbetht »

I can't understand why so little attention is paid for it either..
121312er
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

Just use SpecialFlags=32 with misbehaving servers. It's clearly a server bug if a server reports "Transfer complete", but returns a wrong file size in reply to the SIZE command. The main reason for the SIZE command is to check the size of a file, right?
Author of Total Commander
https://www.ghisler.com
zsero
Junior Member
Junior Member
Posts: 27
Joined: 2007-02-01, 15:03 UTC

Post by *zsero »

Ghisler, it is clearly a server bug but I have it effecting SO MUCH of the servers I know that it would be an important feature to have it on the site-options GUI panel. Something like this:
[] Bypass SIZE check (compatibility with servers reporting false SIZE attribute)
Post Reply