[Csync2] speed of "-cr /"

[Aristedes Maniatis]

On 11/06/10 9:26 PM, Lars Ellenberg wrote:
> Note that this may increase the total runtime,
> if you actually do mean "scan everything",
> I did no benchmarks or anything on that,
> just pointing out the relevant code section....

To bring this to a close.... where the SQL db contains about 100,000 files and running on FreeBSD 8.0 with ZFS on an opteron system a few years old. /usr/ports/packages/ contains a few hundred files.

With the patch:

# time csync2 -cr -G packages /
3.872s

#time csync2 -cr -G packages /usr/ports/packages/
1.085s

I'm guessing that the second one is quicker on the basis that the SQL query returns far fewer records, and so this would be hard to optimise further.



Without the patch:

# time csync2 -cr -G packages /
1m14.022s

#time csync2 -cr -G packages /usr/ports/packages/
1.106s


Clearly this patch has resolved most of the performance problem for this particular use case. If anyone else is wanting to use this on FreeBSD, just create the following file at /usr/ports/net/csync2/files/patch-check.c and then clean and rebuild the port.

--- check.c.orig        2007-07-25 07:04:18.000000000 +1000
+++ check.c     2010-06-12 18:15:46.747200194 +1000
@@ -160,6 +160,10 @@
                         "filename = '%s' %s ORDER BY filename", url_encode(file), where_rec)
         {
                 const char *filename = url_decode(SQL_V[0]);
+
+               if (!csync_match_file(filename))
+                       continue;
+
                 if ( lstat_strict(prefixsubst(filename), &st) != 0 || csync_check_pure(filename) )
                         textlist_add(&tl, filename, 0);
         } SQL_END;



Thanks Lars,


Ari


-- 
-------------------------->
Aristedes Maniatis
ish
http://www.ish.com.au
Level 1, 30 Wilson Street Newtown 2042 Australia
phone +61 2 9550 5001   fax +61 2 9550 4001
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A