Pack from search results: TC passes items multiple times

[white]

(although this optimization is so easy that I would just make it instead of talking about it )?

Cool. How would you do it then? And how would it effect speed and memory usage?

[MVV]

Cool. How would you do it then? And how would it effect speed and memory usage?

Temporary string array or pointer array (if folders are already in some array). Sort, then check if file starts with path from that array using binary search, it is very fast. Binary search allows to compare up to 10 strings (up to first different character) if we have 1000 selected folders.

[ghisler(Author)]

OK, I have added some code now to filter out the duplicates when packing to ZIP, TAR and plugins. Unfortunately it cannot be done with external packers (EXE, e.g. RAR) because TC just passes the directories to them and tells them to recurse. But RAR seems to filter out duplicatss by itself.

Currently when packing tu ZIP, just choose "overwrite all".

[Valentino]

Cool. How would you do it then? And how would it effect speed and memory usage?

Temporary string array or pointer array (if folders are already in some array). Sort, then check if file starts with path from that array using binary search, it is very fast. Binary search allows to compare up to 10 strings (up to first different character) if we have 1000 selected folders.

I don't know how to make it in Delphi, and I don't know what structures TC uses to collect files/folders, but in C++ we can use std::set which stores its items sorted, for example:

Code: Select all

typedef std::set<std::string> StringSet;
StringSet col;
col.insert("F:\\TEMP\\_tc1\\111\\222\\333\\bbb.txt");
col.insert("F:\\TEMP\\_tc1\\111\\222\\333");
col.insert("F:\\TEMP\\_tc1\\111\\222");
col.insert("F:\\TEMP\\_tc1\\111\\222\\444\\ccc.txt");
col.insert("F:\\TEMP\\_tc1\\111\\222\\333\\aaa.txt");

std::cout << "Before:\n\n";
boost::copy(col, std::ostream_iterator<std::string>(std::cout, "\n"));

if (col.size() > 1)
{
   for (StringSet::const_iterator itEnd = col.end(), it = itEnd; --it != col.begin(); )
   {
      if (col.find(GetPath(*it)) != itEnd)
         it = col.erase(it);
   }
}

std::cout << "\nAfter:\n\n";
boost::copy(col, std::ostream_iterator<std::string>(std::cout, "\n"));

[ghisler(Author)]

I was already using a list of objects for the previous items when an archive was modified, so adding this change wasn't too difficult. Please try it!

[MVV]

Seems to be working, thanks!

[ghisler(Author)]

Could you try it both with ZIP and a plugin like 7zip, please?

[MVV]

I tried it with my CopyTree, working. But people say that TC now creates list much slower in case of thousands of files.

Tested it with Windows XP system folder (~6k files), from search results TC takes about 5 seconds to generate filelist. I've used DiskDirExtended plugin for tests and then synced archives, also CopyTree (and compared directories).

[white]

I tested using TC 8.01rc5 32-bit, Windows XP 32-bit, 10000 files of 2k, using internal ZIP.

I created a file list with duplicates. Searched the file list and fed the results to listbox. Then I packed all entries in the listbox.

I did not detect any difference in speed. Packed results are the same as when using TC 8.01rc4.

The removal of duplicates does not always work. It looks like there is a initialization error somewhere.

Sometimes I get overwrite dialogs. Usually only the the first time after I started TC and only one per file.

[ghisler(Author)]

The removal of duplicates does not always work. It looks like there is a initialization error somewhere.

Sorry, I cannot help you without any details.