[Csync2] Csync2 Digest, Vol 62, Issue 1
Tony Muka
tonymuka at gmail.com
Fri May 6 20:50:09 CEST 2011
Wow, thanks for the impressive response time, Lars, i really
appreciate your reply, and test patch! The idea about using the rsync
pattern matching is solid too, although backwards compatibility might
be an issue with config files, so it'd probably need to be switchable.
Unfortunately i'm restricted to using packages available in the Ubuntu
LTS (currently 10.04) repositories and don't have an adequate test
setup at the moment to try this code out.
Thanks again for your hard work!
-tony
On Fri, May 6, 2011 at 6:00 AM, <csync2-request at lists.linbit.com> wrote:
> Send Csync2 mailing list submissions to
> csync2 at lists.linbit.com
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://lists.linbit.com/mailman/listinfo/csync2
> or, via email, send a message with subject or body 'help' to
> csync2-request at lists.linbit.com
>
> You can reach the person managing the list at
> csync2-owner at lists.linbit.com
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Csync2 digest..."
>
>
> Today's Topics:
>
> 1. Re: Partial path excludes? (Tony Muka)
> 2. Re: Partial path excludes? (Lars Ellenberg)
> 3. Re: Partial path excludes? (Lars Ellenberg)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 5 May 2011 11:59:58 -0400
> From: Tony Muka <tonymuka at gmail.com>
> Subject: Re: [Csync2] Partial path excludes?
> To: csync2 at lists.linbit.com
> Message-ID: <BANLkTik0UKkQqAiKyrXOA=jKUe9XvyTmYw at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> Sorry to drag up this old message, but I didn't see much documentation
> on this issue, and wanted to share my findings in trying to exclude
> .svn folders with csync2.
>
> After trying many failing variations of
>
> exclude .svn
> exclude .snv*
> exclude .svn/*
> exclude ".svn/*"
> exclude *svn*
>
> the only thing that worked was
>
> exclude /var/www/*/htdocs/.svn/*;
> exclude /var/www/*/htdocs/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/*/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/*/*/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/*/*/*/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/*/*/*/*/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/*/*/*/*/*/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/*/*/*/*/*/*/*/.svn/*;
>
> It would really be nice if there was a way to use one simple rule to
> exclude the descendants of any .svn folder, but for now, this works.
>
> I probably don't have a good enough reason to exclude the .svn paths
> to use this case as a justification for the feature, etc... Excluding
> the .svn folders does cut my disk usage by 50% on the slave nodes.*
>
>> -----Original Message-----
>> From: csync2-bounces at lists.linbit.com [mailto:csync2-
>> bounces at lists.linbit.com] On Behalf Of Marcus Bointon
>>
>> I have various hierarchies I want to sync, but there are some
>> consistent patterns I want to exclude. From the docs it doesn't sound
>> like I can exclude paths based on wildcards - I can either select
>> explicit paths (/...), or partial filenames (*.php), but not partial
>> paths (somedir/*). The particular case I have is for compiled Smarty
>> templates - essentially anything inside a 'templates_c' directory
>> (anywhere in the hierarchy, and there may be lots that I don't want to
>> have to specify manually) should be ignored. In rsync I do --
>> exclude="templates_c/*". I've tried:
>>
>> exclude templates_c/*
>>
>> but it didn't work (it still synced their contents). How can I do this
>> in csync2?
>
> I have something like that in my csync2.cfg:
>
> exclude /the_base_path_to_my_sites/*/home/www/views_c/*;
>
> It works. However, my case is quite easy: the '/*/' never contains paths but
> always a single dir (say, '/domain.tld/'). I don't know if the '/*/' "thing"
> matches against a path (say, '/this/is/a/customer/path/'), but I think it
> doesn't...
>
> In this case, however, you may try to evince a few patterns in the paths you
> want to exclude, then exclude each of them. Another way could be to
> regularly use some external tool (like a script invoking 'find', in example)
> to discover the paths containing a 'views_c' or 'template_c' directory, then
> automatically add to csync2.cfg an exclude directive for each of these path
> and, finally, csync the csync2.cfg file (this is important: csync2 mail fail
> otherwise!).
>
> Please also note that excluding a dir in a path may impede synchronizing the
> removal of the whole path, because the path will not be empty in the target
> system. So, if you want also to sync the removal of a site, the better thing
> is to avoid 'exclude' in that path. By the way, the kind of exclusion you
> want to do is probably due more to improving synch speed than to a real
> need, isn't it? It may be worth avoiding this optimization if you prefer to
> limit manual adjustments at a minimum.
>
> Giampaolo
>
>
>>
>> Marcus
>> --
>> Marcus Bointon
>> Synchromedia Limited: Creators of http://www.smartmessages.net/
>> UK resellers of info at hand CRM solutions
>> marcus at synchromedia.co.uk | http://www.synchromedia.co.uk/
>>
>
>
> ------------------------------
>
> Message: 2
> Date: Thu, 5 May 2011 21:41:10 +0200
> From: Lars Ellenberg <lars.ellenberg at linbit.com>
> Subject: Re: [Csync2] Partial path excludes?
> To: csync2 at lists.linbit.com
> Message-ID: <20110505194110.GJ4216 at barkeeper1-xen.linbit>
> Content-Type: text/plain; charset=us-ascii
>
> On Thu, May 05, 2011 at 11:59:58AM -0400, Tony Muka wrote:
>> Sorry to drag up this old message, but I didn't see much documentation
>> on this issue, and wanted to share my findings in trying to exclude
>> .svn folders with csync2.
>>
>> After trying many failing variations of
>>
>> exclude .svn
>> exclude .snv*
>> exclude .svn/*
>> exclude ".svn/*"
>> exclude *svn*
>>
>> the only thing that worked was
>>
>> exclude /var/www/*/htdocs/.svn/*;
>> exclude /var/www/*/htdocs/*/.svn/*;
>> exclude /var/www/*/htdocs/*/*/.svn/*;
>> exclude /var/www/*/htdocs/*/*/*/.svn/*;
>> exclude /var/www/*/htdocs/*/*/*/*/.svn/*;
>> exclude /var/www/*/htdocs/*/*/*/*/*/.svn/*;
>> exclude /var/www/*/htdocs/*/*/*/*/*/*/.svn/*;
>> exclude /var/www/*/htdocs/*/*/*/*/*/*/*/.svn/*;
>> exclude /var/www/*/htdocs/*/*/*/*/*/*/*/*/.svn/*;
>> exclude /var/www/*/htdocs/*/*/*/*/*/*/*/*/*/.svn/*;
>>
>> It would really be nice if there was a way to use one simple rule to
>> exclude the descendants of any .svn folder, but for now, this works.
>>
>> I probably don't have a good enough reason to exclude the .svn paths
>> to use this case as a justification for the feature, etc... Excluding
>> the .svn folders does cut my disk usage by 50% on the slave nodes.*
>
> I think that I made a feature request to have that fixed in the 2.0 code
> base, with 2 "**" matching slashes as well.
> I'm currently not sure if that has been done,
> /me checks ...
> Hm. Has not been done :(
>
> Ok, so please do
>
> git clone git://git.linbit.com/csync2.git
>
> apply the quick'n'dirty only compile tested patch below,
> and try
> exclude /var/www/*/htdocs**/.svn;
>
> Note that the "**" is only the trigger, the matching function used is
> still fnmatch, so once the trigger is present, _any_ star in that
> pattern will match slashes (see man fnmatch(3)).
>
> Doing it "right" could be to steal the rsync pattern matching functions,
> and/or to use regexes/pcre, with new keywords exclude-pcre/include-pcre.
>
> Cheers,
>
> Lars
>
>
>
> diff --git a/action.c b/action.c
> index 0328040..9ac8126 100644
> --- a/action.c
> +++ b/action.c
> @@ -44,10 +44,12 @@ void csync_schedule_commands(const char *filename, int islocal)
> continue;
> if (!a->pattern)
> goto found_matching_pattern;
> - for (p=a->pattern; p; p=p->next)
> + for (p=a->pattern; p; p=p->next) {
> + int fnm_pathname = p->star_matches_slashes ? 0 : FNM_PATHNAME;
> if ( !fnmatch(p->pattern, filename,
> - FNM_LEADING_DIR|FNM_PATHNAME) )
> + FNM_LEADING_DIR|fnm_pathname) )
> goto found_matching_pattern;
> + }
> continue;
> found_matching_pattern:
> for (c=a->command; c; c=c->next)
> diff --git a/cfgfile_parser.y b/cfgfile_parser.y
> index 3aadb3e..7f493ab 100644
> --- a/cfgfile_parser.y
> +++ b/cfgfile_parser.y
> @@ -109,12 +109,17 @@ static void add_patt(int patterntype, char *pattern)
> }
> #endif
>
> + /* strip trailing slashes from pattern */
> for (i=strlen(pattern)-1; i>0; i--)
> if (pattern[i] == '/')
> pattern[i] = 0;
> else
> break;
>
> + /* if you use ** at least once anywhere in the pattern,
> + * _all_ stars in the pattern, even single ones,
> + * will match slashes. */
> + t->star_matches_slashes = !!strstr(pattern, "**");
> t->isinclude = patterntype >= 1;
> t->iscompare = patterntype >= 2;
> t->pattern = pattern;
> @@ -283,6 +288,7 @@ static void add_action_pattern(const char *pattern)
> {
> struct csync_group_action_pattern *t =
> calloc(sizeof(struct csync_group_action_pattern), 1);
> + t->star_matches_slashes = !!strstr(pattern, "**");
> t->pattern = pattern;
> t->next = csync_group->action->pattern;
> csync_group->action->pattern = t;
> diff --git a/csync2.h b/csync2.h
> index 5240d15..d76f880 100644
> --- a/csync2.h
> +++ b/csync2.h
> @@ -287,12 +287,13 @@ struct csync_group_host {
>
> struct csync_group_pattern {
> struct csync_group_pattern *next;
> - int isinclude, iscompare;
> + int isinclude, iscompare, star_matches_slashes;
> const char *pattern;
> };
>
> struct csync_group_action_pattern {
> struct csync_group_action_pattern *next;
> + int star_matches_slashes;
> const char *pattern;
> };
>
> diff --git a/groups.c b/groups.c
> index 1ff9a1a..511586e 100644
> --- a/groups.c
> +++ b/groups.c
> @@ -41,8 +41,9 @@ int match_pattern_list(
> matched = 1;
> }
> } else {
> + int fnm_pathname = p->star_matches_slashes ? 0 : FNM_PATHNAME;
> if ( !fnmatch(p->pattern, filename,
> - FNM_LEADING_DIR|FNM_PATHNAME) ) {
> + FNM_LEADING_DIR|fnm_pathname) ) {
> match_path = p->isinclude;
> matched = 1;
> }
> @@ -91,10 +92,11 @@ int csync_step_into(const char *file)
> continue;
> if ( (p->pattern[0] == '/' || p->pattern[0] == '%') && p->isinclude ) {
> char t[strlen(p->pattern)+1], *l;
> + int fnm_pathname = p->star_matches_slashes ? 0 : FNM_PATHNAME;
> strcpy(t, p->pattern);
> while ( (l=strrchr(t, '/')) != 0 ) {
> *l = 0;
> - if ( !fnmatch(t, file, FNM_PATHNAME) )
> + if ( !fnmatch(t, file, fnm_pathname) )
> return 1;
> }
> }
>
> --
> : Lars Ellenberg
> : LINBIT | Your Way to High Availability
> : DRBD/HA support and consulting http://www.linbit.com
>
>
> ------------------------------
>
> Message: 3
> Date: Thu, 5 May 2011 21:59:54 +0200
> From: Lars Ellenberg <lars.ellenberg at linbit.com>
> Subject: Re: [Csync2] Partial path excludes?
> To: csync2 at lists.linbit.com
> Message-ID: <20110505195954.GK4216 at barkeeper1-xen.linbit>
> Content-Type: text/plain; charset=us-ascii
>
> On Thu, May 05, 2011 at 09:41:10PM +0200, Lars Ellenberg wrote:
>> On Thu, May 05, 2011 at 11:59:58AM -0400, Tony Muka wrote:
>> > Sorry to drag up this old message, but I didn't see much documentation
>> > on this issue, and wanted to share my findings in trying to exclude
>> > .svn folders with csync2.
>> >
>> > After trying many failing variations of
>> >
>> > exclude .svn
>> > exclude .snv*
>> > exclude .svn/*
>> > exclude ".svn/*"
>> > exclude *svn*
>> >
>> > the only thing that worked was
>> >
>> > exclude /var/www/*/htdocs/.svn/*;
>> > exclude /var/www/*/htdocs/*/.svn/*;
>> > exclude /var/www/*/htdocs/*/*/.svn/*;
>> > exclude /var/www/*/htdocs/*/*/*/.svn/*;
>> > exclude /var/www/*/htdocs/*/*/*/*/.svn/*;
>> > exclude /var/www/*/htdocs/*/*/*/*/*/.svn/*;
>> > exclude /var/www/*/htdocs/*/*/*/*/*/*/.svn/*;
>> > exclude /var/www/*/htdocs/*/*/*/*/*/*/*/.svn/*;
>> > exclude /var/www/*/htdocs/*/*/*/*/*/*/*/*/.svn/*;
>> > exclude /var/www/*/htdocs/*/*/*/*/*/*/*/*/*/.svn/*;
>> >
>> > It would really be nice if there was a way to use one simple rule to
>> > exclude the descendants of any .svn folder, but for now, this works.
>> >
>> > I probably don't have a good enough reason to exclude the .svn paths
>> > to use this case as a justification for the feature, etc... Excluding
>> > the .svn folders does cut my disk usage by 50% on the slave nodes.*
>>
>> I think that I made a feature request to have that fixed in the 2.0 code
>> base, with 2 "**" matching slashes as well.
>> I'm currently not sure if that has been done,
>> /me checks ...
>> Hm. Has not been done :(
>>
>> Ok, so please do
>>
>> git clone git://git.linbit.com/csync2.git
>>
>> apply the quick'n'dirty only compile tested patch below,
>> and try
>> exclude /var/www/*/htdocs**/.svn;
>>
>> Note that the "**" is only the trigger, the matching function used is
>> still fnmatch, so once the trigger is present, _any_ star in that
>> pattern will match slashes (see man fnmatch(3)).
>>
>> Doing it "right" could be to steal the rsync pattern matching functions,
>
> What I meant here is:
> grab wildmatch.c and wildmatch.h from
> http://gitweb.samba.org/?p=rsync.git;a=tree;f=lib
>
> and use wildmatch(pattern, file)
> where we now use fnmatch().
>
> If someone wants to prepare a patch for that,
> please go ahead ;-)
>
> --
> : Lars Ellenberg
> : LINBIT | Your Way to High Availability
> : DRBD/HA support and consulting http://www.linbit.com
>
>
> ------------------------------
>
> _______________________________________________
> Csync2 mailing list
> Csync2 at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/csync2
>
>
> End of Csync2 Digest, Vol 62, Issue 1
> *************************************
>
More information about the Csync2
mailing list