Advice wanted - extract different files when synchronising

English support forum

Moderators: white, Hacker, petermad, Stefan2

Post Reply
MossMan
Junior Member
Junior Member
Posts: 20
Joined: 2007-03-08, 10:00 UTC

Advice wanted - extract different files when synchronising

Post by *MossMan »

I've been thinking about how to do this for a little while, but can't think of an easy solution... so I'd be happy to hear if anyone has any ideas.

My issue is that I need to periodically synchronise a couple of directories on machines which are on different networks. The way I've solved it is to zip the whole thing on one machine, pass that zip file over the internet, then synchronise from that to the second machine (and if necessary, pass the zip file back to the first machine and synchronise again there).

This works, but the problem is that I'm up/downloading a 250MB file each time to synchronise maybe 10 or 15 files buried in the subdirectories. What I really need is to do the comparison and then extract *only the different files*, so these can be zipped into a couple of MB instead. Crucially, I would still need to preserve the complete paths to the various subdirectories so it's easy to paste them at the other end.

I feel like there must be some trick to do this, but I'm not really sure how...

Hmm... after writing this I have a feeling it would be something like making a copy of the big zip file, filtering on the "=" files and deleting them, leaving all the different ones. I will go do some testing! :)
User avatar
Wilhelm M.
Power Member
Power Member
Posts: 993
Joined: 2003-06-05, 10:45 UTC

Re: Advice wanted - extract different files when synchronising

Post by *Wilhelm M. »

Hello!
I don't if it helps, but I know a tool named "ThirdDir" that compares two directories and copies only the different files to a third directory (hence the name). Since I always use TC for synchronising I have not tested ThirdDir very much, so I cannot tell if the comparison part is really good, but maybe a closer look would not hurt.
Grüße/Regards,
Wilhelm
MossMan
Junior Member
Junior Member
Posts: 20
Joined: 2007-03-08, 10:00 UTC

Re: Advice wanted - extract different files when synchronising

Post by *MossMan »

Ha ha! So yes, I figured it out by posting this topic... Forum therapy!

It's clunky - so I'd be glad to hear about a more elegant method:

- I keep the large zip file on both machines

- I make a copy of it, do a synchronise->comparison against the current local files, filter and delete the "=" files in the zip

- I go through the zip to update each remaining file against the local version (I can't use synchronise for this step as it flags up thousands of missing files!)

- the small zip file is transferred to the other machine and copied over the current files there

- both large zip files are then synchronised against the local files and everything should be in sync

It's a smaller file to transfer, but quite a lot of steps to take - especially if many files have changed. I will try this for a while to see how it goes.
MossMan
Junior Member
Junior Member
Posts: 20
Joined: 2007-03-08, 10:00 UTC

Re: Advice wanted - extract different files when synchronising

Post by *MossMan »

Wilhelm M. wrote: 2020-03-02, 09:55 UTC Hello!
I don't if it helps, but I know a tool named "ThirdDir" that compares two directories and copies only the different files to a third directory (hence the name). Since I always use TC for synchronising I have not tested ThirdDir very much, so I cannot tell if the comparison part is really good, but maybe a closer look would not hurt.
Wow! What a quick reply!

Okay, thanks - I will look into ThirdDir since that sounds like exactly what I need...
MossMan
Junior Member
Junior Member
Posts: 20
Joined: 2007-03-08, 10:00 UTC

Re: Advice wanted - extract different files when synchronising

Post by *MossMan »

Well, it was a nice idea, but as an old tool from 2005(!) it was a bit limited: couldn't read the zip files (I had to unpack them); couldn't handle dayllight savings time (flagged up a bunch of files with a 1 hour offset - TC regards them as identical); didn't preserve the paths when dumping in the third directory (essential for me).

Edit: sorry, it does preserve the full path, I had done two directories and output them to the same third directory.

I'm not sure if it's the DST time issue, but there seem to be identical files appearing in the third directory... Anyway, the limited UI makes it more work than doing it through TC so I'll stick with the method I described.
User avatar
Hacker
Moderator
Moderator
Posts: 13064
Joined: 2003-02-06, 14:56 UTC
Location: Bratislava, Slovakia

Re: Advice wanted - extract different files when synchronising

Post by *Hacker »

MossMan,
Is something like SyncTrayzor out of question? It's like dropbox but without any third party cloud servers, the files get synchronized between all computers directly, peer to peer.

Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.
Post Reply