[Csync2] speed of "-cr /"

[Lars Ellenberg]

On Fri, Jun 11, 2010 at 08:21:00PM +1000, Aristedes Maniatis wrote:
> On 11/06/10 7:44 PM, Lars Ellenberg wrote:
> 
> >First time you mention groups.
> 
> I had not thought they were relevant to the questions I had.
> >but more precise about what you are doing, and included your conf file
> >right away (instead of just one single include line in some followup),
> >you could have been here a few days ago ;-)
> 
> I've deliberately not included a 100 line config file in my original question because I wanted to focus on the specifics of my question.

Right on.
I may have misread your mood.

> >Csync2 does two scans:
> >  it checks for deletions, basically select filenam from file,
> >    is filename still present?  no->  mark as dirty.
> >  it checks for modifications (including additions), basically
> >    scanning directories, comparing with database entries.
> >
> >If you don't explicitly state "-G ", csync2 defaults to all groups.
> >
> >But even if you do "-G single_entry_group", that currently optimizes
> >only the "check for modifications/additions".
> >
> >The "check for deletions" currently does not care for groups.
> 
> I'd go one step further: your check for deletions step doesn't respect
> includes/excludes. Whether that be for a particular group, or for the
> entire configuration (and you haven't run -R recently to clean up lots
> of stale files).

Sure.

> >If different configs had been supported from the very start,
> >groups probably had not been invented in the first place.
> >
> >>I'd rather not create a dozen config files with a dozen databases.
> >
> >Why not?
> 
> Because I'd need a dozen daemons listening on every server.

That won't be necessary, the first thing csync2 communicates is the name
of the config to be used for this connection.

> >Converting your truely independent groups with many files each
> >into their own configs instead seems to be a good idea,
> >even just from a sqlite maintenance, performance, and congestion
> >point of view.
> >
> >I don't see why it would pose an administration overhead, either.
> 
> Personally I like the ease of editing a single file. And it also means
> that I can run a single command to update all the groups, or even just
> pass a list of groups on the command line to they all get executed by
> the one script.

> If I have to worry about multiple listeners, ports and so forth there
> is a little more work to do.

You won't.

> I'll consider it now that I understand the limitations of 'groups' in csync2.

You may also try this patch.  It may, in certain situations (probably
involving prefix substitutions) have a few side-effects I'm not aware
right now, but it should work at least for what you seem to be doing.
Don't forget the explicit "-G ", or it won't have much effect.


diff --git a/check.c b/check.c
index 53736ba..558c514 100644
--- a/check.c
+++ b/check.c
@@ -160,7 +222,12 @@ void csync_check_del(const char *file, int recursive, int init_run)
 			"filename = '%s' %s ORDER BY filename", url_encode(file), where_rec)
 	{
 		const char *filename = url_decode(SQL_V(0));
-		if ( lstat_strict(prefixsubst(filename), &st) != 0 || csync_check_pure(filename) )
+		const char *real_filename = prefixsubst(filename);
+		
+		if (!csync_match_file(real_filename))
+			continue;
+			
+		if ( lstat_strict(real_filename, &st) != 0 || csync_check_pure(filename) )
 			textlist_add(&tl, filename, 0);
 	} SQL_END;
 


-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.