Pack from search results: TC passes items multiple times

Bug reports will be moved here when the described bug has been fixed

Moderators: sheep, Hacker, Stefan2, white

User avatar
white
Power Member
Power Member
Posts: 2020
Joined: 2003-11-19, 08:16 UTC
Location: Netherlands

Post by *white »

Valentino wrote:(although this optimization is so easy that I would just make it instead of talking about it :) )?
Cool. How would you do it then? And how would it effect speed and memory usage?

User avatar
MVV
Power Member
Power Member
Posts: 8486
Joined: 2008-08-03, 12:51 UTC
Location: Russian Federation

Post by *MVV »

white wrote:Cool. How would you do it then? And how would it effect speed and memory usage?
Temporary string array or pointer array (if folders are already in some array). Sort, then check if file starts with path from that array using binary search, it is very fast. Binary search allows to compare up to 10 strings (up to first different character) if we have 1000 selected folders.

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 39604
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

OK, I have added some code now to filter out the duplicates when packing to ZIP, TAR and plugins. Unfortunately it cannot be done with external packers (EXE, e.g. RAR) because TC just passes the directories to them and tells them to recurse. But RAR seems to filter out duplicatss by itself.

Currently when packing tu ZIP, just choose "overwrite all".
Author of Total Commander
http://www.ghisler.com

User avatar
Valentino
Power Member
Power Member
Posts: 702
Joined: 2003-02-07, 00:21 UTC
Location: Ukraine

Post by *Valentino »

MVV wrote:
white wrote:Cool. How would you do it then? And how would it effect speed and memory usage?
Temporary string array or pointer array (if folders are already in some array). Sort, then check if file starts with path from that array using binary search, it is very fast. Binary search allows to compare up to 10 strings (up to first different character) if we have 1000 selected folders.
I don't know how to make it in Delphi, and I don't know what structures TC uses to collect files/folders, but in C++ we can use std::set which stores its items sorted, for example:

Code: Select all

typedef std::set<std::string> StringSet;
StringSet col;
col.insert("F:\\TEMP\\_tc1\\111\\222\\333\\bbb.txt");
col.insert("F:\\TEMP\\_tc1\\111\\222\\333");
col.insert("F:\\TEMP\\_tc1\\111\\222");
col.insert("F:\\TEMP\\_tc1\\111\\222\\444\\ccc.txt");
col.insert("F:\\TEMP\\_tc1\\111\\222\\333\\aaa.txt");

std::cout << "Before:\n\n";
boost::copy(col, std::ostream_iterator<std::string>(std::cout, "\n"));

if (col.size() > 1)
{
   for (StringSet::const_iterator itEnd = col.end(), it = itEnd; --it != col.begin(); )
   {
      if (col.find(GetPath(*it)) != itEnd)
         it = col.erase(it);
   }
}

std::cout << "\nAfter:\n\n";
boost::copy(col, std::ostream_iterator<std::string>(std::cout, "\n"));
Last edited by Valentino on 2012-07-27, 14:47 UTC, edited 1 time in total.

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 39604
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

I was already using a list of objects for the previous items when an archive was modified, so adding this change wasn't too difficult. Please try it!
Author of Total Commander
http://www.ghisler.com

User avatar
MVV
Power Member
Power Member
Posts: 8486
Joined: 2008-08-03, 12:51 UTC
Location: Russian Federation

Post by *MVV »

Seems to be working, thanks! :)

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 39604
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

Could you try it both with ZIP and a plugin like 7zip, please?
Author of Total Commander
http://www.ghisler.com

User avatar
MVV
Power Member
Power Member
Posts: 8486
Joined: 2008-08-03, 12:51 UTC
Location: Russian Federation

Post by *MVV »

I tried it with my CopyTree, working. But people say that TC now creates list much slower in case of thousands of files.

Tested it with Windows XP system folder (~6k files), from search results TC takes about 5 seconds to generate filelist. I've used DiskDirExtended plugin for tests and then synced archives, also CopyTree (and compared directories).

User avatar
white
Power Member
Power Member
Posts: 2020
Joined: 2003-11-19, 08:16 UTC
Location: Netherlands

Post by *white »

I tested using TC 8.01rc5 32-bit, Windows XP 32-bit, 10000 files of 2k, using internal ZIP.

I created a file list with duplicates. Searched the file list and fed the results to listbox. Then I packed all entries in the listbox.

I did not detect any difference in speed. Packed results are the same as when using TC 8.01rc4.

The removal of duplicates does not always work. It looks like there is a initialization error somewhere.

Sometimes I get overwrite dialogs. Usually only the the first time after I started TC and only one per file.

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 39604
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

The removal of duplicates does not always work. It looks like there is a initialization error somewhere.
Sorry, I cannot help you without any details.
Author of Total Commander
http://www.ghisler.com

Post Reply