File disappear after copy from server to local

Bug reports will be moved here when the described bug has been fixed

Moderators: white, sheep, Hacker, Stefan2

Post Reply
User avatar
Feriman
Junior Member
Junior Member
Posts: 5
Joined: 2021-09-23, 08:16 UTC
Contact:

File disappear after copy from server to local

Post by *Feriman »

Hi,

I use sftpplug/SFTP plugin with the latest version of TC (10.00), and I can reproduce this annyoing bug.

The bug: If the remote connection disconnected OR the remote file has been removed, and you copy the already copied file to local, you will lost the local file.

Steps to reproduce the bug:

1., Open Total Commander with already installed sftpplug plugin
2., Open one server inside of Network/Secure FTP on left side
3., Copy any file from the server (left side) to the local machine (right side)
4., Delete this file from the server (DONT through the Total Commander, use console instead)
5., Copy this file again from the server (left side) to the local machine (right side)
6., Error message appear, then the local file disappear

As I remember I can reproduce this bug in TC 9.5x as well, but not sure.

Can you do the same?


Feriman
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 41696
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: File disappear after copy from server to local

Post by *ghisler(Author) »

Well, that cannot be avoided - something similar happens when the connection is lost in the middle of a transfer. Then you will have a partially downloaded file which has overwritten the local file.
Author of Total Commander
http://www.ghisler.com
User avatar
Feriman
Junior Member
Junior Member
Posts: 5
Joined: 2021-09-23, 08:16 UTC
Contact:

Re: File disappear after copy from server to local

Post by *Feriman »

Well, I though on the possible solution, and I found two way. They are compatible with each other.

1., Check the connection and file avalibility before starting the copy process. If one of them failed, let's do nothing, just print the error message (Connection lost or file/folder is not available anymore).

2., Copy file into temp file. If the connection lost, remove this temp file. If successful, move this temp file to overwrite the original one. Another possible solution is delete the file into Recycle bin, and restore it from here if connection lost. If successful, you can remove this file from the Recycle bin.
User avatar
Dalai
Power Member
Power Member
Posts: 7825
Joined: 2005-01-28, 22:17 UTC
Location: Meiningen (Südthüringen)

Re: File disappear after copy from server to local

Post by *Dalai »

Feriman wrote: 2021-11-03, 05:50 UTC1., Check the connection and file avalibility before starting the copy process. If one of them failed, let's do nothing, just print the error message (Connection lost or file/folder is not available anymore).
That's what I was going to suggest. Just check if the file is still there before doing the transfer (overwriting the file), and abort if it's not.
2., Copy file into temp file. If the connection lost, remove this temp file. If successful, move this temp file to overwrite the original one. Another possible solution is delete the file into Recycle bin, and restore it from here if connection lost. If successful, you can remove this file from the Recycle bin.
That won't work, when
  • the temp directory has not enough free space left for the file in question, or even runs out mid-transfer
  • the destination is a network (UNC) share/drive because such locations don't have a recycle bin
As for the last bit of this suggestion: Are applications able to delete files from the recycle bin selectively, is there an API function for this? Emphasis on selectively here.

I see another possible solution: Rename the target file before initiating the transfer and delete the renamed file only after successful transfer. But this also has its drawbacks:
  • if the target file has special permissions set, they are lost.
  • it requires some space on the destination, likely double the size of the transferred file (maybe even more), just like the way above that goes via recycle bin.
I recommend to manually refresh the source directory right before doing the transfer, and perhaps rename the target file beforehand. Sure, this has its own drawbacks, mainly in the area of convenience.

Regards
Dalai
#101164 Personal licence
Ryzen 5 2600, 16 GiB RAM, ASUS Prime X370-A, Win7 x64

Plugins: Services2, Startups - Download-Mirror
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 41696
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: File disappear after copy from server to local

Post by *ghisler(Author) »

I have already considered the idea of the TEMP file, but the problem is that you wouldn't inherit the permissions of the overwritten file - it would be as if there had been no local file at all.

Checking for the presence of the remote file for each single download is also a problem:
Method 1: Initiate the download, and once it has started, try to create the local file and prompt the user for overwriting. This is a problem if the user doesn't react quickly enough (connection can time out), or the local file cannot be opened for writing (e.g. locked by another app), so we now need to somehow abort the operation.
Method 2: Make an extra call to the server before each download just to check whether the file is still there. This would be slower, and wouldn't ensure that the remote file can actually be opened.

Currently I'm considering a third method, where I open the target file for writing, but don't truncate it yet. This is a different CreateFile option, so I will have to check first whether it has any negative effects. For example, the opening could work but the truncation could fail when there are partial locks in the middle of the file (e.g. a database).
Author of Total Commander
http://www.ghisler.com
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 41696
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: File disappear after copy from server to local

Post by *ghisler(Author) »

I have added method three now to the SFTP plugin -> new version 2.90:
https://www.ghisler.com/plugins.htm#filesys

Please let me know what you think. I do exactly what I suggested above:
1. Open the target file for writing, but don't truncate it yet.
2. Start the download
3a. Once we have received at least one byte, truncate the target
3b. If the download fails, keep the target unchanged

Please let me know what you think! The Cloud and Webdav plugins use the same (old) logic, so I will modify them too if you think that it's a good solution.
Author of Total Commander
http://www.ghisler.com
User avatar
Feriman
Junior Member
Junior Member
Posts: 5
Joined: 2021-09-23, 08:16 UTC
Contact:

Re: File disappear after copy from server to local

Post by *Feriman »

Nice job, works as expected! Thank you for fixing this bug!
Post Reply