Consolidating music-archives with SyncDirs

[HalbschuhTouri]

When trying to consolidate some music-archives scattered over various USB-sticks recently I again ran into the somewhat frustrating situation that TC apparently doesn't seem well equipped to remedy such a - indeed quite typical - file-management-operation.

Problem was that many files had been renamed following slightly changed/evolving naming-conventions over time and so I had to try to consolidate the resulting dupes by somehow comparing all those different versions by binary content in order to eliminate dupes following old-naming-convention-styles and finally consolidating the remaining unique versions on a new large and dedicated USB-drive.

Now FileFind would offer the ability to compare files by binary content - but this apparently would only work if all the candidates for comparison had been located below a conjoint parent Dir on the same drive. Furthermore - even if FileFind-duplicate-search would somehow work over separate Dirs/drives - its paired representation of the dupes found (one by one) would hardly be suited to handle any revision-oriented file-operations thereafter (copying, renaming, deleting) in a group-wise, directory-/album-oriented manner.

So my next idea was that the SyncDirs-dialog would seem to be ideally suited to handle that latter situation as whole parent-directories or even sub-trees in a grouped-together fashion can be selected for copying/moving/deleting therein.

Yet unfortunately, in spite of having checked the "by content"-checkbox offered within the SyncDirs-process-dialog, I couldn't succeed with that approach either as it turned out that all of these obvious dupes comprised in that heap would turn up as either "unique left" or "unique right" because of their (perhaps only slightly) different names and in spite of their equal size.

So is there absolutely no way to compare files by content (as that checkbox would suggest), regardless of their file-name (as in FileFind dupe-search) within the SyncDirs-process? Or did I miss something here and there is a better way to handle that problem within TC?

[petermad]

2HalbschuhTouri

Maybe you could use the Multi-rename tool to rename the files with old convention names to the new convention (and thereafter use the Synchronization tool to eliminate files with the same content).

If the old files are all named after the same convention, that should be doable, but if they are named with several different conventions, it can be problematic or impossible.

[Dalai]

Now FileFind would offer the ability to compare files by binary content - but this apparently would only work if all the candidates for comparison had been located below a conjoint parent Dir on the same drive.

No, TC's search can search in multiple directories and thus also find duplicates in them. Just enter all paths or drives separated by semi-colon into the "Search in" field.

So is there absolutely no way to compare files by content (as that checkbox would suggest), regardless of their file-name (as in FileFind dupe-search) within the SyncDirs-process?

Primary criterion for the Synchronize directories is the filename, all other factors like date, size and content are lower-ranking. Synchronize directories' goal is to do just that: bring two directory trees into sync. It doesn't deal with (multiple) duplicates in the same directory tree or across the two trees (unless they have the exact same relative path).

Regards
Dalai

[HalbschuhTouri]

Maybe you could use the Multi-rename tool to rename the files with old convention names to the new convention (and thereafter use the Synchronization tool to eliminate files with the same content).

If the old files are all named after the same convention, that should be doable, but if they are named with several different conventions, it can be problematic or impossible.

Sounds like good advice but unfortunately
1. the latter condition is true and there are several "conventions" - or should I better say: "pseudo-conventions" as the old titles found do not always adhere strictly to the respective convention at the time but might also contain some unwanted remnants from the original filenames when downloading which do not happen to be of any systematic nature at all and -
2. to make things worse - those old names at times will also contain some mis-spellings of words or garbled German-Umlauts or Spanish special characters etc., fully or partly corrected in some versions and still in their original mis-spelled form in others.

In short: a systematic, let alone fully automatic renaming-process towards the currently desired (as well as also correctly spelled) convention-compliant name seems completely out of reach. As I'm currently overlooking the problem there will be no way around a binary comparison by content while at the same time disregarding the file-name. Because this seems the only way I'd be able to eliminate 80% of these old-style/mis-spelled file-names by identifying them as dupes and eliminating them altogether without the hassle of having to "streamline" their file-names on an individual basis first. Being forced to "manually" having to rename perhaps thousands of files first before I could SyncDirs let identify many hundreds among them (perhaps up to 85%) as dupes sounds like an absolute deal-breaker.

On the other hand even one single mis-spelled letter in one of the file-names under review, e.g. "Elvis Pres(s)ley" or "Emmy(i)lou Harris" or with a blank/underscore between their names, will render SyncDirs totally incapacitated and defunct when it comes to identifying binary dupes leaving both name-versions stand as "unique left/right" in some rather misleading fashion thereby thwarting any attempts of reconciliation.

Is there perhaps - to your knowledge or anybody else's in this forum - some other dedicated software-tool around outside TC that can do better in remedying such a situation?

[HalbschuhTouri]

No, TC's search can search in multiple directories and thus also find duplicates in them. Just enter all paths or drives separated by semi-colon into the "Search in" field.

Could you please elaborate. Do I really have to type in the different starting-paths manually here? I do remember some TC-enhancement-suggestion in the corresponding TC-forum some months ago - which I also supported then - that proposed to simply set the starting-paths for such a search between different Dirs by selecting them as active and inactive main-panel in TC each and then to offer some way in FileFind to inherit those paths from both the main-panels of TC.

Simple and effective as well as IMHO very easy to implement - and yet this proposal went down without any answer or feedback from Mr. Ghisler, our cherished Author of TC, at that time. So I suppose this still cannot be done in current TC, including the presently offered beta-versions of TC11? Or is there another way to interactively enter those starting-paths for the search via the GUI short of having to type them in manually?

[Dalai]

You can copy/paste the paths to make it a bit easier.

Suggestions for new features very often take time to implement. Keep in mind that TC's betas are almost feature-complete which means that larger changes won't get implemented because they would most likely introduce new bugs and/or inconsistencies.

Regards
Dalai

[Sir_SiLvA]

Sounds like good advice but...

And dont forget that they could also differ in the content of the (mp3)tags so even a binary compare is not always usefull, but
if you wanna do a quick binary compare with automic deletion of doubles i would suggest https://clonespy.com/

[HalbschuhTouri]

You can copy/paste the paths to make it a bit easier.

Suggestions for new features very often take time to implement. Keep in mind that TC's betas are almost feature-complete which means that larger changes won't get implemented because they would most likely introduce new bugs and/or inconsistencies.

Well, copy/paste was actually pretty much what I had in mind when talking about "manual input". Tedious! Speaking of which - pasting looks relatively simple while copying maybe not so. What would be the simplest way to copy the current path of the active panel? <Ctrl><Shift><Enter> to copy the current path to the command-line and then <Ctrl>C from there? Clumsy! Is there a direct shortcut that I'm unaware of?

And concerning the implementation of new features into betas: I suppose that is exactly why this proposal has been introduced around or after Christmas IIRC, well before the current beta so as to avoid any hurry caused by late implementation under pressure and thereby leading to an error-prone outcome. But it looks like it may not even have been noticed then - that said, of course it is finally Mr. Ghisler's call to decide what proposals to implement and which to dismiss.

But those above described shortcomings in both FileFind and SyncDirs (the latter ones of which you seemingly consider as intentional and without any improvement needed) are well known for over a decade now and no really significant improvements have been introduced since then in this particular respect of allowing users to consolidate folders in spite of both of these tools already showing tremendous potential for expansion/enhancement of their current capabilities by looking at both concepts in a more generalized way which could perfectly also include the ability to effectively consolidate folders in the end. Since - of all tools - which one if not a powerful file-manager should be capable of handling even such a complex yet by no means uncommon task like consolidation with excellence? I would say: "It's about time!" and I really have a hard time perceiving any elements of "a hurry" in this moderate request.

[HalbschuhTouri]

And dont forget that they could also differ in the content of the (mp3)tags so even a binary compare is not always usefull, but
if you wanna do a quick binary compare with automic deletion of doubles i would suggest https://clonespy.com/

I do not use ".mp3"-format because of its lossy nature - only ".flac" and ".ape" or ".wav" to convert. And since those differently named dupes all stem from the same initial downloads and the actual audio hasn't been altered since then there should be no problems with different tags either. As for "clonespy" - thanks for the tip. I'll have a look into it!

[Dalai]

What would be the simplest way to copy the current path of the active panel? <Ctrl><Shift><Enter> to copy the current path to the command-line and then <Ctrl>C from there? Clumsy! Is there a direct shortcut that I'm unaware of?

The cm_CopySrcPathToClip command can be put on a button (which I have done for my TC installations) and/or a hotkey via TC's Options > Misc. You can also make a slow double-click on the path bar (to switch it to edit mode) and copy the path from there.

I suppose that is exactly why this proposal has been introduced around or after Christmas IIRC, well before the current beta

Well, suggestions may take months or even years to be implemented. Recently I checked if one of my suggestions which I made in 2016 had been implemented - and it didn't surprise me that nothing changed. Sometimes people need time to think about a suggestion and how to implement it - all while working on other things.

Regards
Dalai

[petermad]

2HalbschuhTouri

type in the different starting-paths manually

Can't you just start the search higher up in the directory tree? - The search will take a longer time, but the result will be more or less the same, since only files with the same size will be found. Restrict the search to only relevant file types (*.mp3).

There is a well hidden feature in TC for searching for duplicates - after the search is done, then Feed it to Listbox. Then press + on the numeric keyboard, and a special dialog for selecting duplicates pops up:

https://tcmd.madsenworld.dk/findduplicates.png
https://tcmd.madsenworld.dk/selectduplicates1.png
https://tcmd.madsenworld.dk/selectduplicates2.png
https://tcmd.madsenworld.dk/selectduplicateshelp.png

You can use the "Select duplicat files" dialog to select all the older versions and then delete them, and then use the "Synchronize directories" tool to copy the new versions to the directories where the older versions were removed from.

[HalbschuhTouri]

The cm_CopySrcPathToClip command can be put on a button (which I have done for my TC installations) and/or a hotkey via TC's Options > Misc. You can also make a slow double-click on the path bar (to switch it to edit mode) and copy the path from there.

Thanks a lot, @Dalai , for that hint. That button is in the making as it clearly represents the least clumsy method for the intended Task. IMHO this command should be a standard-menu-entry in TC combined with a standard-shortcut.

[Sir_SiLvA]

IMHO this command should be a standard-menu-entry in TC combined with a standard-shortcut.

Thats why I created my Menu in the first place I mean the last time the Menu got an overhaul was only 29(!) years ago...
but dont worry I am working on the translation of my extended Menu with all commands to be ready soon

[HalbschuhTouri]

Can't you just start the search higher up in the directory tree? - The search will take a longer time, but the result will be more or less the same, since only files with the same size will be found. Restrict the search to only relevant file types (*.mp3).

That - while very time-consuming on multi-TB-drives - would only be an option if those old archives were not scattered across several different drives/USB-sticks where each USB-stick (at least the newer ones) happens to be a 1TB-drive itself and in contrast the older, smaller ones are considerably slower concerning their reading-/writing-speed.

There is a well hidden feature in TC for searching for duplicates - after the search is done, then Feed it to Listbox. Then press + on the numeric keyboard, and a special dialog for selecting duplicates pops up:

https://tcmd.madsenworld.dk/findduplicates.png
https://tcmd.madsenworld.dk/selectduplicates1.png
https://tcmd.madsenworld.dk/selectduplicates2.png
https://tcmd.madsenworld.dk/selectduplicateshelp.png

With that hint you have somehow hit the nail on its head. I'm only aware of this feature for about half a year now while having used TC for decades. It is precisely the (even more hidden) "Select by folder"-tab that - at least partly - does the trick. And I have never understood why such a powerful feature would practically run with exclusion of the public. I always considered myself being a power-user of TC and to a degree still do - but I've been really unaware of that capability for (possibly) decades - as presumably still are 99% of regular TC-users.

And yet - having said all that - as compared to the potential of the Sync-Dirs-dialog this feature is not a perfect solution either. Because - other than the SyncDirs group-wise representation of files grouped below their parent folder - this "Select by folder"-tab will not show the contents of the folders it offers for selection at the same time (as SyncDir does). Hence - in particular when there may be dozens of folders listed and offered for selection - the user will again be left in uncertainty about which folders to choose in the end.

[petermad]

Hence - in particular when there may be dozens of folders listed and offered for selection - the user will again be left in uncertainty about which folders to choose in the end.

Remember that you can still see those suggested folders in TC's main panel in the Feed to Listbox view.

those old archives were not scattered across several different drives/USB-sticks where each USB-stick

I see. Maybe it is time for you with a little house cleaning, moving all those archives together in a common folder