From tonymuka at gmail.com  Thu May  5 17:59:58 2011
From: tonymuka at gmail.com (Tony Muka)
Date: Thu, 5 May 2011 11:59:58 -0400
Subject: [Csync2] Partial path excludes?
Message-ID: <BANLkTik0UKkQqAiKyrXOA=jKUe9XvyTmYw@mail.gmail.com>

Sorry to drag up this old message, but I didn't see much documentation
on this issue, and wanted to share my findings in trying to exclude
.svn folders with csync2.

After trying many failing variations of

exclude .svn
exclude .snv*
exclude .svn/*
exclude ".svn/*"
exclude *svn*

the only thing that worked was

exclude /var/www/*/htdocs/.svn/*;
exclude /var/www/*/htdocs/*/.svn/*;
exclude /var/www/*/htdocs/*/*/.svn/*;
exclude /var/www/*/htdocs/*/*/*/.svn/*;
exclude /var/www/*/htdocs/*/*/*/*/.svn/*;
exclude /var/www/*/htdocs/*/*/*/*/*/.svn/*;
exclude /var/www/*/htdocs/*/*/*/*/*/*/.svn/*;
exclude /var/www/*/htdocs/*/*/*/*/*/*/*/.svn/*;
exclude /var/www/*/htdocs/*/*/*/*/*/*/*/*/.svn/*;
exclude /var/www/*/htdocs/*/*/*/*/*/*/*/*/*/.svn/*;

It would really be nice if there was a way to use one simple rule to
exclude the descendants of any .svn folder, but for now, this works.

I probably don't have a good enough reason to exclude the .svn paths
to use this case as a justification for the feature, etc... Excluding
the .svn folders does cut my disk usage by 50% on the slave nodes.*

> -----Original Message-----
> From: csync2-bounces at lists.linbit.com [mailto:csync2-
> bounces at lists.linbit.com] On Behalf Of Marcus Bointon
>
> I have various hierarchies I want to sync, but there are some
> consistent patterns I want to exclude. From the docs it doesn't sound
> like I can exclude paths based on wildcards - I can either select
> explicit paths (/...), or partial filenames (*.php), but not partial
> paths (somedir/*). The particular case I have is for compiled Smarty
> templates - essentially anything inside a 'templates_c' directory
> (anywhere in the hierarchy, and there may be lots that I don't want to
> have to specify manually) should be ignored. In rsync I do --
> exclude="templates_c/*". I've tried:
>
> exclude templates_c/*
>
> but it didn't work (it still synced their contents). How can I do this
> in csync2?

I have something like that in my csync2.cfg:

	exclude /the_base_path_to_my_sites/*/home/www/views_c/*;

It works. However, my case is quite easy: the '/*/' never contains paths but
always a single dir (say, '/domain.tld/'). I don't know if the '/*/' "thing"
matches against a path (say, '/this/is/a/customer/path/'), but I think it
doesn't...

In this case, however, you may try to evince a few patterns in the paths you
want to exclude, then exclude each of them. Another way could be to
regularly use some external tool (like a script invoking 'find', in example)
to discover the paths containing a 'views_c' or 'template_c' directory, then
automatically add to csync2.cfg an exclude directive for each of these path
and, finally, csync the csync2.cfg file (this is important: csync2 mail fail
otherwise!).

Please also note that excluding a dir in a path may impede synchronizing the
removal of the whole path, because the path will not be empty in the target
system. So, if you want also to sync the removal of a site, the better thing
is to avoid 'exclude' in that path. By the way, the kind of exclusion you
want to do is probably due more to improving synch speed than to a real
need, isn't it? It may be worth avoiding this optimization if you prefer to
limit manual adjustments at a minimum.

Giampaolo


>
> Marcus
> --
> Marcus Bointon
> Synchromedia Limited: Creators of http://www.smartmessages.net/
> UK resellers of info at hand CRM solutions
> marcus at synchromedia.co.uk | http://www.synchromedia.co.uk/
>

From lars.ellenberg at linbit.com  Thu May  5 21:41:10 2011
From: lars.ellenberg at linbit.com (Lars Ellenberg)
Date: Thu, 5 May 2011 21:41:10 +0200
Subject: [Csync2] Partial path excludes?
In-Reply-To: <BANLkTik0UKkQqAiKyrXOA=jKUe9XvyTmYw@mail.gmail.com>
References: <BANLkTik0UKkQqAiKyrXOA=jKUe9XvyTmYw@mail.gmail.com>
Message-ID: <20110505194110.GJ4216@barkeeper1-xen.linbit>

On Thu, May 05, 2011 at 11:59:58AM -0400, Tony Muka wrote:
> Sorry to drag up this old message, but I didn't see much documentation
> on this issue, and wanted to share my findings in trying to exclude
> .svn folders with csync2.
> 
> After trying many failing variations of
> 
> exclude .svn
> exclude .snv*
> exclude .svn/*
> exclude ".svn/*"
> exclude *svn*
> 
> the only thing that worked was
> 
> exclude /var/www/*/htdocs/.svn/*;
> exclude /var/www/*/htdocs/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/*/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/*/*/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/*/*/*/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/*/*/*/*/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/*/*/*/*/*/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/*/*/*/*/*/*/*/.svn/*;
> 
> It would really be nice if there was a way to use one simple rule to
> exclude the descendants of any .svn folder, but for now, this works.
> 
> I probably don't have a good enough reason to exclude the .svn paths
> to use this case as a justification for the feature, etc... Excluding
> the .svn folders does cut my disk usage by 50% on the slave nodes.*

I think that I made a feature request to have that fixed in the 2.0 code
base, with 2 "**" matching slashes as well.
I'm currently not sure if that has been done,
 /me checks ...
Hm. Has not been done :(

Ok, so please do

  git clone git://git.linbit.com/csync2.git

apply the quick'n'dirty only compile tested patch below,
and try
	exclude /var/www/*/htdocs**/.svn;

Note that the "**" is only the trigger, the matching function used is
still fnmatch, so once the trigger is present, _any_ star in that
pattern will match slashes (see man fnmatch(3)).

Doing it "right" could be to steal the rsync pattern matching functions,
and/or to use regexes/pcre, with new keywords exclude-pcre/include-pcre.

Cheers,

	Lars



diff --git a/action.c b/action.c
index 0328040..9ac8126 100644
--- a/action.c
+++ b/action.c
@@ -44,10 +44,12 @@ void csync_schedule_commands(const char *filename, int islocal)
 				continue;
 			if (!a->pattern)
 				goto found_matching_pattern;
-			for (p=a->pattern; p; p=p->next)
+			for (p=a->pattern; p; p=p->next) {
+				int fnm_pathname = p->star_matches_slashes ? 0 : FNM_PATHNAME;
 				if ( !fnmatch(p->pattern, filename,
-						FNM_LEADING_DIR|FNM_PATHNAME) )
+						FNM_LEADING_DIR|fnm_pathname) )
 					goto found_matching_pattern;
+			}
 			continue;
 found_matching_pattern:
 			for (c=a->command; c; c=c->next)
diff --git a/cfgfile_parser.y b/cfgfile_parser.y
index 3aadb3e..7f493ab 100644
--- a/cfgfile_parser.y
+++ b/cfgfile_parser.y
@@ -109,12 +109,17 @@ static void add_patt(int patterntype, char *pattern)
 	}
 #endif
 
+	/* strip trailing slashes from pattern */
 	for (i=strlen(pattern)-1; i>0; i--)
 		if (pattern[i] == '/')
 			pattern[i] = 0;
 		else
 			break;
 
+	/* if you use ** at least once anywhere in the pattern,
+	 * _all_ stars in the pattern, even single ones,
+	 * will match slashes. */
+	t->star_matches_slashes = !!strstr(pattern, "**");
 	t->isinclude = patterntype >= 1;
 	t->iscompare = patterntype >= 2;
 	t->pattern = pattern;
@@ -283,6 +288,7 @@ static void add_action_pattern(const char *pattern)
 {
 	struct csync_group_action_pattern *t =
 		calloc(sizeof(struct csync_group_action_pattern), 1);
+	t->star_matches_slashes = !!strstr(pattern, "**");
 	t->pattern = pattern;
 	t->next = csync_group->action->pattern;
 	csync_group->action->pattern = t;
diff --git a/csync2.h b/csync2.h
index 5240d15..d76f880 100644
--- a/csync2.h
+++ b/csync2.h
@@ -287,12 +287,13 @@ struct csync_group_host {
 
 struct csync_group_pattern {
 	struct csync_group_pattern *next;
-	int isinclude, iscompare;
+	int isinclude, iscompare, star_matches_slashes;
 	const char *pattern;
 };
 
 struct csync_group_action_pattern {
 	struct csync_group_action_pattern *next;
+	int star_matches_slashes;
 	const char *pattern;
 };
 
diff --git a/groups.c b/groups.c
index 1ff9a1a..511586e 100644
--- a/groups.c
+++ b/groups.c
@@ -41,8 +41,9 @@ int match_pattern_list(
 				matched = 1;
 			}
 		} else {
+			int fnm_pathname = p->star_matches_slashes ? 0 : FNM_PATHNAME;
 			if ( !fnmatch(p->pattern, filename,
-					FNM_LEADING_DIR|FNM_PATHNAME) ) {
+					FNM_LEADING_DIR|fnm_pathname) ) {
 				match_path = p->isinclude;
 				matched = 1;
 			}
@@ -91,10 +92,11 @@ int csync_step_into(const char *file)
 				continue;
 			if ( (p->pattern[0] == '/' || p->pattern[0] == '%') && p->isinclude ) {
 				char t[strlen(p->pattern)+1], *l;
+				int fnm_pathname = p->star_matches_slashes ? 0 : FNM_PATHNAME;
 				strcpy(t, p->pattern);
 				while ( (l=strrchr(t, '/')) != 0 ) {
 					*l = 0;
-					if ( !fnmatch(t, file, FNM_PATHNAME) )
+					if ( !fnmatch(t, file, fnm_pathname) )
 								return 1;
 				}
 			}

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

From lars.ellenberg at linbit.com  Thu May  5 21:59:54 2011
From: lars.ellenberg at linbit.com (Lars Ellenberg)
Date: Thu, 5 May 2011 21:59:54 +0200
Subject: [Csync2] Partial path excludes?
In-Reply-To: <20110505194110.GJ4216@barkeeper1-xen.linbit>
References: <BANLkTik0UKkQqAiKyrXOA=jKUe9XvyTmYw@mail.gmail.com>
	<20110505194110.GJ4216@barkeeper1-xen.linbit>
Message-ID: <20110505195954.GK4216@barkeeper1-xen.linbit>

On Thu, May 05, 2011 at 09:41:10PM +0200, Lars Ellenberg wrote:
> On Thu, May 05, 2011 at 11:59:58AM -0400, Tony Muka wrote:
> > Sorry to drag up this old message, but I didn't see much documentation
> > on this issue, and wanted to share my findings in trying to exclude
> > .svn folders with csync2.
> > 
> > After trying many failing variations of
> > 
> > exclude .svn
> > exclude .snv*
> > exclude .svn/*
> > exclude ".svn/*"
> > exclude *svn*
> > 
> > the only thing that worked was
> > 
> > exclude /var/www/*/htdocs/.svn/*;
> > exclude /var/www/*/htdocs/*/.svn/*;
> > exclude /var/www/*/htdocs/*/*/.svn/*;
> > exclude /var/www/*/htdocs/*/*/*/.svn/*;
> > exclude /var/www/*/htdocs/*/*/*/*/.svn/*;
> > exclude /var/www/*/htdocs/*/*/*/*/*/.svn/*;
> > exclude /var/www/*/htdocs/*/*/*/*/*/*/.svn/*;
> > exclude /var/www/*/htdocs/*/*/*/*/*/*/*/.svn/*;
> > exclude /var/www/*/htdocs/*/*/*/*/*/*/*/*/.svn/*;
> > exclude /var/www/*/htdocs/*/*/*/*/*/*/*/*/*/.svn/*;
> > 
> > It would really be nice if there was a way to use one simple rule to
> > exclude the descendants of any .svn folder, but for now, this works.
> > 
> > I probably don't have a good enough reason to exclude the .svn paths
> > to use this case as a justification for the feature, etc... Excluding
> > the .svn folders does cut my disk usage by 50% on the slave nodes.*
> 
> I think that I made a feature request to have that fixed in the 2.0 code
> base, with 2 "**" matching slashes as well.
> I'm currently not sure if that has been done,
>  /me checks ...
> Hm. Has not been done :(
> 
> Ok, so please do
> 
>   git clone git://git.linbit.com/csync2.git
> 
> apply the quick'n'dirty only compile tested patch below,
> and try
> 	exclude /var/www/*/htdocs**/.svn;
> 
> Note that the "**" is only the trigger, the matching function used is
> still fnmatch, so once the trigger is present, _any_ star in that
> pattern will match slashes (see man fnmatch(3)).
> 
> Doing it "right" could be to steal the rsync pattern matching functions,

What I meant here is:
grab wildmatch.c and wildmatch.h from
http://gitweb.samba.org/?p=rsync.git;a=tree;f=lib

and use wildmatch(pattern, file)
where we now use fnmatch().

If someone wants to prepare a patch for that,
please go ahead ;-)

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

From tonymuka at gmail.com  Fri May  6 20:50:09 2011
From: tonymuka at gmail.com (Tony Muka)
Date: Fri, 6 May 2011 14:50:09 -0400
Subject: [Csync2] Csync2 Digest, Vol 62, Issue 1
In-Reply-To: <mailman.1.1304676005.343.csync2@lists.linbit.com>
References: <mailman.1.1304676005.343.csync2@lists.linbit.com>
Message-ID: <BANLkTimfhd=azcowsAcOEof7unKHOeBUyg@mail.gmail.com>

Wow, thanks for the impressive response time, Lars, i really
appreciate your reply, and test patch! The idea about using the rsync
pattern matching is solid too, although backwards compatibility might
be an issue with config files, so it'd probably need to be switchable.

Unfortunately i'm restricted to using packages available in the Ubuntu
LTS (currently 10.04) repositories and don't have an adequate test
setup at the moment to try this code out.

Thanks again for your hard work!
-tony

On Fri, May 6, 2011 at 6:00 AM,  <csync2-request at lists.linbit.com> wrote:
> Send Csync2 mailing list submissions to
> ? ? ? ?csync2 at lists.linbit.com
>
> To subscribe or unsubscribe via the World Wide Web, visit
> ? ? ? ?http://lists.linbit.com/mailman/listinfo/csync2
> or, via email, send a message with subject or body 'help' to
> ? ? ? ?csync2-request at lists.linbit.com
>
> You can reach the person managing the list at
> ? ? ? ?csync2-owner at lists.linbit.com
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Csync2 digest..."
>
>
> Today's Topics:
>
> ? 1. Re: Partial path excludes? (Tony Muka)
> ? 2. Re: Partial path excludes? (Lars Ellenberg)
> ? 3. Re: Partial path excludes? (Lars Ellenberg)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 5 May 2011 11:59:58 -0400
> From: Tony Muka <tonymuka at gmail.com>
> Subject: Re: [Csync2] Partial path excludes?
> To: csync2 at lists.linbit.com
> Message-ID: <BANLkTik0UKkQqAiKyrXOA=jKUe9XvyTmYw at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> Sorry to drag up this old message, but I didn't see much documentation
> on this issue, and wanted to share my findings in trying to exclude
> .svn folders with csync2.
>
> After trying many failing variations of
>
> exclude .svn
> exclude .snv*
> exclude .svn/*
> exclude ".svn/*"
> exclude *svn*
>
> the only thing that worked was
>
> exclude /var/www/*/htdocs/.svn/*;
> exclude /var/www/*/htdocs/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/*/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/*/*/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/*/*/*/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/*/*/*/*/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/*/*/*/*/*/*/.svn/*;
> exclude /var/www/*/htdocs/*/*/*/*/*/*/*/*/*/.svn/*;
>
> It would really be nice if there was a way to use one simple rule to
> exclude the descendants of any .svn folder, but for now, this works.
>
> I probably don't have a good enough reason to exclude the .svn paths
> to use this case as a justification for the feature, etc... Excluding
> the .svn folders does cut my disk usage by 50% on the slave nodes.*
>
>> -----Original Message-----
>> From: csync2-bounces at lists.linbit.com [mailto:csync2-
>> bounces at lists.linbit.com] On Behalf Of Marcus Bointon
>>
>> I have various hierarchies I want to sync, but there are some
>> consistent patterns I want to exclude. From the docs it doesn't sound
>> like I can exclude paths based on wildcards - I can either select
>> explicit paths (/...), or partial filenames (*.php), but not partial
>> paths (somedir/*). The particular case I have is for compiled Smarty
>> templates - essentially anything inside a 'templates_c' directory
>> (anywhere in the hierarchy, and there may be lots that I don't want to
>> have to specify manually) should be ignored. In rsync I do --
>> exclude="templates_c/*". I've tried:
>>
>> exclude templates_c/*
>>
>> but it didn't work (it still synced their contents). How can I do this
>> in csync2?
>
> I have something like that in my csync2.cfg:
>
> ? ? ? ?exclude /the_base_path_to_my_sites/*/home/www/views_c/*;
>
> It works. However, my case is quite easy: the '/*/' never contains paths but
> always a single dir (say, '/domain.tld/'). I don't know if the '/*/' "thing"
> matches against a path (say, '/this/is/a/customer/path/'), but I think it
> doesn't...
>
> In this case, however, you may try to evince a few patterns in the paths you
> want to exclude, then exclude each of them. Another way could be to
> regularly use some external tool (like a script invoking 'find', in example)
> to discover the paths containing a 'views_c' or 'template_c' directory, then
> automatically add to csync2.cfg an exclude directive for each of these path
> and, finally, csync the csync2.cfg file (this is important: csync2 mail fail
> otherwise!).
>
> Please also note that excluding a dir in a path may impede synchronizing the
> removal of the whole path, because the path will not be empty in the target
> system. So, if you want also to sync the removal of a site, the better thing
> is to avoid 'exclude' in that path. By the way, the kind of exclusion you
> want to do is probably due more to improving synch speed than to a real
> need, isn't it? It may be worth avoiding this optimization if you prefer to
> limit manual adjustments at a minimum.
>
> Giampaolo
>
>
>>
>> Marcus
>> --
>> Marcus Bointon
>> Synchromedia Limited: Creators of http://www.smartmessages.net/
>> UK resellers of info at hand CRM solutions
>> marcus at synchromedia.co.uk | http://www.synchromedia.co.uk/
>>
>
>
> ------------------------------
>
> Message: 2
> Date: Thu, 5 May 2011 21:41:10 +0200
> From: Lars Ellenberg <lars.ellenberg at linbit.com>
> Subject: Re: [Csync2] Partial path excludes?
> To: csync2 at lists.linbit.com
> Message-ID: <20110505194110.GJ4216 at barkeeper1-xen.linbit>
> Content-Type: text/plain; charset=us-ascii
>
> On Thu, May 05, 2011 at 11:59:58AM -0400, Tony Muka wrote:
>> Sorry to drag up this old message, but I didn't see much documentation
>> on this issue, and wanted to share my findings in trying to exclude
>> .svn folders with csync2.
>>
>> After trying many failing variations of
>>
>> exclude .svn
>> exclude .snv*
>> exclude .svn/*
>> exclude ".svn/*"
>> exclude *svn*
>>
>> the only thing that worked was
>>
>> exclude /var/www/*/htdocs/.svn/*;
>> exclude /var/www/*/htdocs/*/.svn/*;
>> exclude /var/www/*/htdocs/*/*/.svn/*;
>> exclude /var/www/*/htdocs/*/*/*/.svn/*;
>> exclude /var/www/*/htdocs/*/*/*/*/.svn/*;
>> exclude /var/www/*/htdocs/*/*/*/*/*/.svn/*;
>> exclude /var/www/*/htdocs/*/*/*/*/*/*/.svn/*;
>> exclude /var/www/*/htdocs/*/*/*/*/*/*/*/.svn/*;
>> exclude /var/www/*/htdocs/*/*/*/*/*/*/*/*/.svn/*;
>> exclude /var/www/*/htdocs/*/*/*/*/*/*/*/*/*/.svn/*;
>>
>> It would really be nice if there was a way to use one simple rule to
>> exclude the descendants of any .svn folder, but for now, this works.
>>
>> I probably don't have a good enough reason to exclude the .svn paths
>> to use this case as a justification for the feature, etc... Excluding
>> the .svn folders does cut my disk usage by 50% on the slave nodes.*
>
> I think that I made a feature request to have that fixed in the 2.0 code
> base, with 2 "**" matching slashes as well.
> I'm currently not sure if that has been done,
> ?/me checks ...
> Hm. Has not been done :(
>
> Ok, so please do
>
> ?git clone git://git.linbit.com/csync2.git
>
> apply the quick'n'dirty only compile tested patch below,
> and try
> ? ? ? ?exclude /var/www/*/htdocs**/.svn;
>
> Note that the "**" is only the trigger, the matching function used is
> still fnmatch, so once the trigger is present, _any_ star in that
> pattern will match slashes (see man fnmatch(3)).
>
> Doing it "right" could be to steal the rsync pattern matching functions,
> and/or to use regexes/pcre, with new keywords exclude-pcre/include-pcre.
>
> Cheers,
>
> ? ? ? ?Lars
>
>
>
> diff --git a/action.c b/action.c
> index 0328040..9ac8126 100644
> --- a/action.c
> +++ b/action.c
> @@ -44,10 +44,12 @@ void csync_schedule_commands(const char *filename, int islocal)
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?continue;
> ? ? ? ? ? ? ? ? ? ? ? ?if (!a->pattern)
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?goto found_matching_pattern;
> - ? ? ? ? ? ? ? ? ? ? ? for (p=a->pattern; p; p=p->next)
> + ? ? ? ? ? ? ? ? ? ? ? for (p=a->pattern; p; p=p->next) {
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? int fnm_pathname = p->star_matches_slashes ? 0 : FNM_PATHNAME;
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?if ( !fnmatch(p->pattern, filename,
> - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? FNM_LEADING_DIR|FNM_PATHNAME) )
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? FNM_LEADING_DIR|fnm_pathname) )
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?goto found_matching_pattern;
> + ? ? ? ? ? ? ? ? ? ? ? }
> ? ? ? ? ? ? ? ? ? ? ? ?continue;
> ?found_matching_pattern:
> ? ? ? ? ? ? ? ? ? ? ? ?for (c=a->command; c; c=c->next)
> diff --git a/cfgfile_parser.y b/cfgfile_parser.y
> index 3aadb3e..7f493ab 100644
> --- a/cfgfile_parser.y
> +++ b/cfgfile_parser.y
> @@ -109,12 +109,17 @@ static void add_patt(int patterntype, char *pattern)
> ? ? ? ?}
> ?#endif
>
> + ? ? ? /* strip trailing slashes from pattern */
> ? ? ? ?for (i=strlen(pattern)-1; i>0; i--)
> ? ? ? ? ? ? ? ?if (pattern[i] == '/')
> ? ? ? ? ? ? ? ? ? ? ? ?pattern[i] = 0;
> ? ? ? ? ? ? ? ?else
> ? ? ? ? ? ? ? ? ? ? ? ?break;
>
> + ? ? ? /* if you use ** at least once anywhere in the pattern,
> + ? ? ? ?* _all_ stars in the pattern, even single ones,
> + ? ? ? ?* will match slashes. */
> + ? ? ? t->star_matches_slashes = !!strstr(pattern, "**");
> ? ? ? ?t->isinclude = patterntype >= 1;
> ? ? ? ?t->iscompare = patterntype >= 2;
> ? ? ? ?t->pattern = pattern;
> @@ -283,6 +288,7 @@ static void add_action_pattern(const char *pattern)
> ?{
> ? ? ? ?struct csync_group_action_pattern *t =
> ? ? ? ? ? ? ? ?calloc(sizeof(struct csync_group_action_pattern), 1);
> + ? ? ? t->star_matches_slashes = !!strstr(pattern, "**");
> ? ? ? ?t->pattern = pattern;
> ? ? ? ?t->next = csync_group->action->pattern;
> ? ? ? ?csync_group->action->pattern = t;
> diff --git a/csync2.h b/csync2.h
> index 5240d15..d76f880 100644
> --- a/csync2.h
> +++ b/csync2.h
> @@ -287,12 +287,13 @@ struct csync_group_host {
>
> ?struct csync_group_pattern {
> ? ? ? ?struct csync_group_pattern *next;
> - ? ? ? int isinclude, iscompare;
> + ? ? ? int isinclude, iscompare, star_matches_slashes;
> ? ? ? ?const char *pattern;
> ?};
>
> ?struct csync_group_action_pattern {
> ? ? ? ?struct csync_group_action_pattern *next;
> + ? ? ? int star_matches_slashes;
> ? ? ? ?const char *pattern;
> ?};
>
> diff --git a/groups.c b/groups.c
> index 1ff9a1a..511586e 100644
> --- a/groups.c
> +++ b/groups.c
> @@ -41,8 +41,9 @@ int match_pattern_list(
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?matched = 1;
> ? ? ? ? ? ? ? ? ? ? ? ?}
> ? ? ? ? ? ? ? ?} else {
> + ? ? ? ? ? ? ? ? ? ? ? int fnm_pathname = p->star_matches_slashes ? 0 : FNM_PATHNAME;
> ? ? ? ? ? ? ? ? ? ? ? ?if ( !fnmatch(p->pattern, filename,
> - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? FNM_LEADING_DIR|FNM_PATHNAME) ) {
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? FNM_LEADING_DIR|fnm_pathname) ) {
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?match_path = p->isinclude;
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?matched = 1;
> ? ? ? ? ? ? ? ? ? ? ? ?}
> @@ -91,10 +92,11 @@ int csync_step_into(const char *file)
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?continue;
> ? ? ? ? ? ? ? ? ? ? ? ?if ( (p->pattern[0] == '/' || p->pattern[0] == '%') && p->isinclude ) {
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?char t[strlen(p->pattern)+1], *l;
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? int fnm_pathname = p->star_matches_slashes ? 0 : FNM_PATHNAME;
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?strcpy(t, p->pattern);
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?while ( (l=strrchr(t, '/')) != 0 ) {
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?*l = 0;
> - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? if ( !fnmatch(t, file, FNM_PATHNAME) )
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? if ( !fnmatch(t, file, fnm_pathname) )
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?return 1;
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?}
> ? ? ? ? ? ? ? ? ? ? ? ?}
>
> --
> : Lars Ellenberg
> : LINBIT | Your Way to High Availability
> : DRBD/HA support and consulting http://www.linbit.com
>
>
> ------------------------------
>
> Message: 3
> Date: Thu, 5 May 2011 21:59:54 +0200
> From: Lars Ellenberg <lars.ellenberg at linbit.com>
> Subject: Re: [Csync2] Partial path excludes?
> To: csync2 at lists.linbit.com
> Message-ID: <20110505195954.GK4216 at barkeeper1-xen.linbit>
> Content-Type: text/plain; charset=us-ascii
>
> On Thu, May 05, 2011 at 09:41:10PM +0200, Lars Ellenberg wrote:
>> On Thu, May 05, 2011 at 11:59:58AM -0400, Tony Muka wrote:
>> > Sorry to drag up this old message, but I didn't see much documentation
>> > on this issue, and wanted to share my findings in trying to exclude
>> > .svn folders with csync2.
>> >
>> > After trying many failing variations of
>> >
>> > exclude .svn
>> > exclude .snv*
>> > exclude .svn/*
>> > exclude ".svn/*"
>> > exclude *svn*
>> >
>> > the only thing that worked was
>> >
>> > exclude /var/www/*/htdocs/.svn/*;
>> > exclude /var/www/*/htdocs/*/.svn/*;
>> > exclude /var/www/*/htdocs/*/*/.svn/*;
>> > exclude /var/www/*/htdocs/*/*/*/.svn/*;
>> > exclude /var/www/*/htdocs/*/*/*/*/.svn/*;
>> > exclude /var/www/*/htdocs/*/*/*/*/*/.svn/*;
>> > exclude /var/www/*/htdocs/*/*/*/*/*/*/.svn/*;
>> > exclude /var/www/*/htdocs/*/*/*/*/*/*/*/.svn/*;
>> > exclude /var/www/*/htdocs/*/*/*/*/*/*/*/*/.svn/*;
>> > exclude /var/www/*/htdocs/*/*/*/*/*/*/*/*/*/.svn/*;
>> >
>> > It would really be nice if there was a way to use one simple rule to
>> > exclude the descendants of any .svn folder, but for now, this works.
>> >
>> > I probably don't have a good enough reason to exclude the .svn paths
>> > to use this case as a justification for the feature, etc... Excluding
>> > the .svn folders does cut my disk usage by 50% on the slave nodes.*
>>
>> I think that I made a feature request to have that fixed in the 2.0 code
>> base, with 2 "**" matching slashes as well.
>> I'm currently not sure if that has been done,
>> ?/me checks ...
>> Hm. Has not been done :(
>>
>> Ok, so please do
>>
>> ? git clone git://git.linbit.com/csync2.git
>>
>> apply the quick'n'dirty only compile tested patch below,
>> and try
>> ? ? ? exclude /var/www/*/htdocs**/.svn;
>>
>> Note that the "**" is only the trigger, the matching function used is
>> still fnmatch, so once the trigger is present, _any_ star in that
>> pattern will match slashes (see man fnmatch(3)).
>>
>> Doing it "right" could be to steal the rsync pattern matching functions,
>
> What I meant here is:
> grab wildmatch.c and wildmatch.h from
> http://gitweb.samba.org/?p=rsync.git;a=tree;f=lib
>
> and use wildmatch(pattern, file)
> where we now use fnmatch().
>
> If someone wants to prepare a patch for that,
> please go ahead ;-)
>
> --
> : Lars Ellenberg
> : LINBIT | Your Way to High Availability
> : DRBD/HA support and consulting http://www.linbit.com
>
>
> ------------------------------
>
> _______________________________________________
> Csync2 mailing list
> Csync2 at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/csync2
>
>
> End of Csync2 Digest, Vol 62, Issue 1
> *************************************
>

From Tim.Cheyne at gen-i.co.nz  Sat May  7 06:54:56 2011
From: Tim.Cheyne at gen-i.co.nz (Tim Cheyne)
Date: Sat, 7 May 2011 16:54:56 +1200
Subject: [Csync2] csync2 -m with prefix
Message-ID: <CD9C1C62902348438FAD1C366EB4B3AC1A43EE9868@AKEXMBX01.telecom.tcnz.net>

Hi,
trying to mark files as dirty with csync -m but they don't always display with csync2 -M.
I believe it only works when the directory is not a prefix but having difficulty trying to specify a prefix
on the command line e.g.

/etc/csync2/csync2_lists.cfg has prefix defined

prefix rbldhome
{
        on *:   /var/lib/rbldns/local/working;
}

csync -m /tmp/txfer/* -C lists                        works

csync - m /var/lib/rbldns/local/working/thefile -C lists     not working
csync - m %rbldns%/thefile -C lists             not working
csync - m \%rbldns\%/thefile -C lists                 not working
csync - m "%rbldns%/thefile" -C lists                 not working

If you cannot specify a prefix on the command line does the directory have to be in a pattern?

Thanks for any help
Cheers
Tim


Mgate3.telecom.co.nz made the following annotations
---------------------------------------------------------------------
This communication, including any attachments, is confidential. If you are not the intended recipient, you should not read it - please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002.
---------------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/csync2/attachments/20110507/97daed97/attachment-0001.htm>

From sherwood.mcgowan at gmail.com  Sat May  7 12:40:24 2011
From: sherwood.mcgowan at gmail.com (Sherwood McGowan)
Date: Sat, 7 May 2011 05:40:24 -0500
Subject: [Csync2] possible to force csync2 to use a specific interface?
Message-ID: <BANLkTim30qo3BknSkmW5aJ+AWuvHmnsxrQ@mail.gmail.com>

Gentlemen,

I've done quite a bit of googling and have not been able to find anything
regarding the ability to force csync2 to use a specific interface. Is it
really possible? I've tried using the hostname at interface configuration
option as well as calling csync2 with the -N hostname option, but
identification still fails...

It works normally, but I have a virtual interface set on eth0:0 (only 1 NIC
on the servers) and since the address on eth0:0 does not match the hostname
being provided (because eth0 "has" that hostname), I keep getting
failures...

Thoughts? Suggestions?

Thanks in advance!


-- 
Sherwood McGowan
Telecommunications and VOIP Consultant
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/csync2/attachments/20110507/21169a3e/attachment.htm>

From infos at e-blokos.com  Sat May  7 16:47:43 2011
From: infos at e-blokos.com (E-Blokos)
Date: Sat, 7 May 2011 10:47:43 -0400
Subject: [Csync2] possible to force csync2 to use a specific interface?
References: <BANLkTim30qo3BknSkmW5aJ+AWuvHmnsxrQ@mail.gmail.com>
Message-ID: <836CB1BE3F534D4D8376C3B85B0F9FCB@e1705>

if you are on linux doesn't matter interface if
you have your hostname in /etc/hosts
  ----- Original Message ----- 
  From: Sherwood McGowan 
  To: csync2 at lists.linbit.com 
  Sent: Saturday, May 07, 2011 6:40 AM
  Subject: [Csync2] possible to force csync2 to use a specific interface?


  Gentlemen,

  I've done quite a bit of googling and have not been able to find anything regarding the ability to force csync2 to use a specific interface. Is it really possible? I've tried using the hostname at interface configuration option as well as calling csync2 with the -N hostname option, but identification still fails...

  It works normally, but I have a virtual interface set on eth0:0 (only 1 NIC on the servers) and since the address on eth0:0 does not match the hostname being provided (because eth0 "has" that hostname), I keep getting failures...

  Thoughts? Suggestions?

  Thanks in advance!


  -- 
  Sherwood McGowan
  Telecommunications and VOIP Consultant




------------------------------------------------------------------------------


  _______________________________________________
  Csync2 mailing list
  Csync2 at lists.linbit.com
  http://lists.linbit.com/mailman/listinfo/csync2
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/csync2/attachments/20110507/03e48cf8/attachment.htm>

From dennis at schafroth.dk  Sun May 15 15:59:47 2011
From: dennis at schafroth.dk (Dennis Schafroth)
Date: Sun, 15 May 2011 15:59:47 +0200
Subject: [Csync2] Regression in 2.0-rc1?
Message-ID: <76B02B70-914E-4B23-B738-6F294D542A8A@schafroth.dk>

Hi 

I started using this yesterday, but is seeing a lot of errors: the log from the sending side looks like this:

May 15 05:52:23 zotac csync2: ERROR: Could not open result from tempnam(/export/home/dennis/Projects/csync2/trunk/.urlencode.c.27198)!
May 15 05:52:23 zotac csync2: I/O Error 'No such file or directory' in rsync-check: /export/home/dennis/Projects/csync2/trunk/urlencode.c
May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/update.c): bluebox0.schafroth.dk octet-stream 36
May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/stamp-h1): bluebox0.schafroth.dk rs#0016
May 15 05:52:23 zotac csync2: ERROR: Could not open result from tempnam(/export/home/dennis/Projects/csync2/trunk/.rsync.c.XXXXXX)!
May 15 05:52:23 zotac csync2: I/O Error 'No such file or directory' in rsync-check: /export/home/dennis/Projects/csync2/trunk/rsync.c
May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/release.sh): bluebox0.schafroth.dk octet-stream 180
May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/prefixsubst.c): bluebox0.schafroth.dk rs#0016
May 15 05:52:23 zotac csync2: ERROR: Could not open result from tempnam(/export/home/dennis/Projects/csync2/trunk/.newdb.txt.XXXXXX)!
May 15 05:52:23 zotac csync2: I/O Error 'No such file or directory' in rsync-check: /export/home/dennis/Projects/csync2/trunk/newdb.txt
May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/mkinstalldirs): bluebox0.schafroth.dk octet-stream 24
May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/missing): bluebox0.schafroth.dk rs#0016
May 15 05:52:23 zotac csync2: ERROR: Could not open result from tempnam(/export/home/dennis/Projects/csync2/trunk/.logger.c.XXXXXX)!
May 15 05:52:23 zotac csync2: I/O Error 'No such file or directory' in rsync-check: /export/home/dennis/Projects/csync2/trunk/logger.c
May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/ipc.h): bluebox0.schafroth.dk octet-stream 84
May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/ipc.c): bluebox0.schafroth.dk rs#0016
May 15 05:52:23 zotac csync2: ERROR: Could not open result from tempnam(/export/home/dennis/Projects/csync2/trunk/.ipc-send.c.XXXXXX)!
May 15 05:52:23 zotac csync2: I/O Error 'No such file or directory' in rsync-check: /export/home/dennis/Projects/csync2/trunk/ipc-send.c
May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/ipc-listen.c): bluebox0.schafroth.dk octet-stream 36
May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/install-sh): bluebox0.schafroth.dk rs#0016
May 15 05:52:23 zotac csync2: ERROR: Could not open result from tempnam(/export/home/dennis/Projects/csync2/trunk/.groups.c.XXXXXX)!
May 15 05:52:23 zotac csync2: I/O Error 'No such file or directory' in rsync-check: /export/home/dennis/Projects/csync2/trunk/groups.c
May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/getrealfn.c): bluebox0.schafroth.dk octet-stream 36

And the command terminates premature with a "Format-error while receiving data." 

Am I the only one seeing this?  

the third string print in the "ERROR from peer" looks pretty much like random data:

May 15 05:52:24 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/cygwin/.svn/text-base/readme_pkg.txt.svn-base): bluebox0.schafroth.dk #004l<CA><F4><A8>#013<A4>??OK (cmd_finished).

There is also an (old) issue with backup when uses prefixes, but I have a patch for that coming up. 

cheers, 
:-Dennis

From genady.puzis at xeround.com  Sun May 15 16:10:04 2011
From: genady.puzis at xeround.com (Genady Puzis)
Date: Sun, 15 May 2011 17:10:04 +0300
Subject: [Csync2] Csync and soft links
Message-ID: <4DCFDEBC.1010004@xeround.com>

Hi,

my csync doesn't follow symbolic links.
Is it normal behavior or should i possibly add some flags or config options?

Thanks,
Genady

From lars.ellenberg at linbit.com  Mon May 16 21:55:57 2011
From: lars.ellenberg at linbit.com (Lars Ellenberg)
Date: Mon, 16 May 2011 21:55:57 +0200
Subject: [Csync2] Regression in 2.0-rc1?
In-Reply-To: <76B02B70-914E-4B23-B738-6F294D542A8A@schafroth.dk>
References: <76B02B70-914E-4B23-B738-6F294D542A8A@schafroth.dk>
Message-ID: <20110516195557.GD1870@barkeeper1-xen.linbit>

On Sun, May 15, 2011 at 03:59:47PM +0200, Dennis Schafroth wrote:
> Hi 
> 
> I started using this yesterday, but is seeing a lot of errors: the log from the sending side looks like this:

rc1, or current git?
standalone csync2, or via inetd?

> May 15 05:52:23 zotac csync2: ERROR: Could not open result from tempnam(/export/home/dennis/Projects/csync2/trunk/.urlencode.c.27198)!
> May 15 05:52:23 zotac csync2: I/O Error 'No such file or directory' in rsync-check: /export/home/dennis/Projects/csync2/trunk/urlencode.c
> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/update.c): bluebox0.schafroth.dk octet-stream 36

Only with %prefix%es, or always?

> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/stamp-h1): bluebox0.schafroth.dk rs#0016
> May 15 05:52:23 zotac csync2: ERROR: Could not open result from tempnam(/export/home/dennis/Projects/csync2/trunk/.rsync.c.XXXXXX)!
> May 15 05:52:23 zotac csync2: I/O Error 'No such file or directory' in rsync-check: /export/home/dennis/Projects/csync2/trunk/rsync.c
> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/release.sh): bluebox0.schafroth.dk octet-stream 180
> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/prefixsubst.c): bluebox0.schafroth.dk rs#0016
> May 15 05:52:23 zotac csync2: ERROR: Could not open result from tempnam(/export/home/dennis/Projects/csync2/trunk/.newdb.txt.XXXXXX)!
> May 15 05:52:23 zotac csync2: I/O Error 'No such file or directory' in rsync-check: /export/home/dennis/Projects/csync2/trunk/newdb.txt
> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/mkinstalldirs): bluebox0.schafroth.dk octet-stream 24
> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/missing): bluebox0.schafroth.dk rs#0016
> May 15 05:52:23 zotac csync2: ERROR: Could not open result from tempnam(/export/home/dennis/Projects/csync2/trunk/.logger.c.XXXXXX)!
> May 15 05:52:23 zotac csync2: I/O Error 'No such file or directory' in rsync-check: /export/home/dennis/Projects/csync2/trunk/logger.c
> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/ipc.h): bluebox0.schafroth.dk octet-stream 84
> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/ipc.c): bluebox0.schafroth.dk rs#0016
> May 15 05:52:23 zotac csync2: ERROR: Could not open result from tempnam(/export/home/dennis/Projects/csync2/trunk/.ipc-send.c.XXXXXX)!
> May 15 05:52:23 zotac csync2: I/O Error 'No such file or directory' in rsync-check: /export/home/dennis/Projects/csync2/trunk/ipc-send.c
> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/ipc-listen.c): bluebox0.schafroth.dk octet-stream 36
> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/install-sh): bluebox0.schafroth.dk rs#0016
> May 15 05:52:23 zotac csync2: ERROR: Could not open result from tempnam(/export/home/dennis/Projects/csync2/trunk/.groups.c.XXXXXX)!
> May 15 05:52:23 zotac csync2: I/O Error 'No such file or directory' in rsync-check: /export/home/dennis/Projects/csync2/trunk/groups.c
> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/getrealfn.c): bluebox0.schafroth.dk octet-stream 36
> 
> And the command terminates premature with a "Format-error while receiving data." 
> 
> Am I the only one seeing this?  
> 
> the third string print in the "ERROR from peer" looks pretty much like random data:
> 
> May 15 05:52:24 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/cygwin/.svn/text-base/readme_pkg.txt.svn-base): bluebox0.schafroth.dk #004l<CA><F4><A8>#013<A4>??OK (cmd_finished).
> 
> There is also an (old) issue with backup when uses prefixes, but I have a patch for that coming up. 

Thanks,

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD? and LINBIT? are registered trademarks of LINBIT, Austria.

From lars.ellenberg at linbit.com  Mon May 16 21:56:52 2011
From: lars.ellenberg at linbit.com (Lars Ellenberg)
Date: Mon, 16 May 2011 21:56:52 +0200
Subject: [Csync2] Csync and soft links
In-Reply-To: <4DCFDEBC.1010004@xeround.com>
References: <4DCFDEBC.1010004@xeround.com>
Message-ID: <20110516195652.GE1870@barkeeper1-xen.linbit>

On Sun, May 15, 2011 at 05:10:04PM +0300, Genady Puzis wrote:
> Hi,
> 
> my csync doesn't follow symbolic links.
> Is it normal behavior or should i possibly add some flags or config options?

That's expected and normal behaviour,
though I admit that it can be annoying at times.

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD? and LINBIT? are registered trademarks of LINBIT, Austria.

From dennis at schafroth.dk  Tue May 17 10:16:21 2011
From: dennis at schafroth.dk (Dennis Schafroth)
Date: Tue, 17 May 2011 10:16:21 +0200
Subject: [Csync2] Regression in 2.0-rc1?
In-Reply-To: <20110516195557.GD1870@barkeeper1-xen.linbit>
References: <76B02B70-914E-4B23-B738-6F294D542A8A@schafroth.dk>
	<20110516195557.GD1870@barkeeper1-xen.linbit>
Message-ID: <30415343-6810-4EF5-AAF5-9DAE8377EC3B@schafroth.dk>


On 16/05/2011, at 21.55, Lars Ellenberg wrote:

> On Sun, May 15, 2011 at 03:59:47PM +0200, Dennis Schafroth wrote:
>> Hi 
>> 
>> I started using this yesterday, but is seeing a lot of errors: the log from the sending side looks like this:
> 
> rc1, or current git?

Git. Did a pull. Didnt see any comming down: 

commit b7490b4d2f9069e4e364c9f4824956ade4bad691
Author: Lars Ellenberg <lars at linbit.com>
Date:   Wed Feb 9 13:24:51 2011 +0000

    fix compiler warning about empty snprintf format


> standalone csync2, or via inetd?

inetd. 

Since then I have tried running in standalone to debug it, but haven't been able to reproduce. So yeah, been thinking in the direction of inetd issue. 

> 
>> May 15 05:52:23 zotac csync2: ERROR: Could not open result from tempnam(/export/home/dennis/Projects/csync2/trunk/.urlencode.c.27198)!
>> May 15 05:52:23 zotac csync2: I/O Error 'No such file or directory' in rsync-check: /export/home/dennis/Projects/csync2/trunk/urlencode.c
>> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/update.c): bluebox0.schafroth.dk octet-stream 36
> 
>  Only with %prefix%es, or always?

Only remember seeing it with prefixes.
> 
>> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/stamp-h1): bluebox0.schafroth.dk rs#0016
>> May 15 05:52:23 zotac csync2: ERROR: Could not open result from tempnam(/export/home/dennis/Projects/csync2/trunk/.rsync.c.XXXXXX)!
>> May 15 05:52:23 zotac csync2: I/O Error 'No such file or directory' in rsync-check: /export/home/dennis/Projects/csync2/trunk/rsync.c
>> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/release.sh): bluebox0.schafroth.dk octet-stream 180
>> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/prefixsubst.c): bluebox0.schafroth.dk rs#0016
>> May 15 05:52:23 zotac csync2: ERROR: Could not open result from tempnam(/export/home/dennis/Projects/csync2/trunk/.newdb.txt.XXXXXX)!
>> May 15 05:52:23 zotac csync2: I/O Error 'No such file or directory' in rsync-check: /export/home/dennis/Projects/csync2/trunk/newdb.txt
>> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/mkinstalldirs): bluebox0.schafroth.dk octet-stream 24
>> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/missing): bluebox0.schafroth.dk rs#0016
>> May 15 05:52:23 zotac csync2: ERROR: Could not open result from tempnam(/export/home/dennis/Projects/csync2/trunk/.logger.c.XXXXXX)!
>> May 15 05:52:23 zotac csync2: I/O Error 'No such file or directory' in rsync-check: /export/home/dennis/Projects/csync2/trunk/logger.c
>> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/ipc.h): bluebox0.schafroth.dk octet-stream 84
>> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/ipc.c): bluebox0.schafroth.dk rs#0016
>> May 15 05:52:23 zotac csync2: ERROR: Could not open result from tempnam(/export/home/dennis/Projects/csync2/trunk/.ipc-send.c.XXXXXX)!
>> May 15 05:52:23 zotac csync2: I/O Error 'No such file or directory' in rsync-check: /export/home/dennis/Projects/csync2/trunk/ipc-send.c
>> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/ipc-listen.c): bluebox0.schafroth.dk octet-stream 36
>> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/install-sh): bluebox0.schafroth.dk rs#0016
>> May 15 05:52:23 zotac csync2: ERROR: Could not open result from tempnam(/export/home/dennis/Projects/csync2/trunk/.groups.c.XXXXXX)!
>> May 15 05:52:23 zotac csync2: I/O Error 'No such file or directory' in rsync-check: /export/home/dennis/Projects/csync2/trunk/groups.c
>> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/getrealfn.c): bluebox0.schafroth.dk octet-stream 36
>> 
>> And the command terminates premature with a "Format-error while receiving data." 
>> 
>> Am I the only one seeing this?  
>> 
>> the third string print in the "ERROR from peer" looks pretty much like random data:
>> 
>> May 15 05:52:24 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/cygwin/.svn/text-base/readme_pkg.txt.svn-base): bluebox0.schafroth.dk #004l<CA><F4><A8>#013<A4>??OK (cmd_finished).
>> 
>> There is also an (old) issue with backup when uses prefixes, but I have a patch for that coming up. 
> 
> Thanks,
> 
> -- 
> : Lars Ellenberg
> : LINBIT | Your Way to High Availability
> : DRBD/HA support and consulting http://www.linbit.com
> 
> DRBD? and LINBIT? are registered trademarks of LINBIT, Austria.
> _______________________________________________
> Csync2 mailing list
> Csync2 at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/csync2


From lars.ellenberg at linbit.com  Tue May 17 13:24:34 2011
From: lars.ellenberg at linbit.com (Lars Ellenberg)
Date: Tue, 17 May 2011 13:24:34 +0200
Subject: [Csync2] Regression in 2.0-rc1?
In-Reply-To: <30415343-6810-4EF5-AAF5-9DAE8377EC3B@schafroth.dk>
References: <76B02B70-914E-4B23-B738-6F294D542A8A@schafroth.dk>
	<20110516195557.GD1870@barkeeper1-xen.linbit>
	<30415343-6810-4EF5-AAF5-9DAE8377EC3B@schafroth.dk>
Message-ID: <20110517112434.GA7717@barkeeper1-xen.linbit>

On Tue, May 17, 2011 at 10:16:21AM +0200, Dennis Schafroth wrote:
> 
> On 16/05/2011, at 21.55, Lars Ellenberg wrote:
> 
> > On Sun, May 15, 2011 at 03:59:47PM +0200, Dennis Schafroth wrote:
> >> Hi 
> >> 
> >> I started using this yesterday, but is seeing a lot of errors: the log from the sending side looks like this:
> > 
> > rc1, or current git?
> 
> Git. Did a pull. Didnt see any comming down: 
> 
> commit b7490b4d2f9069e4e364c9f4824956ade4bad691

> > standalone csync2, or via inetd?
> 
> inetd. 
> 
> Since then I have tried running in standalone to debug it, but haven't been able to reproduce. So yeah, been thinking in the direction of inetd issue. 

If you can give me a short reproducer,
I'll see to it that it gets fixed.

> >  Only with %prefix%es, or always?
> 
> Only remember seeing it with prefixes.

> >> May 15 05:52:23 zotac csync2: I/O Error 'No such file or directory' in rsync-check: /export/home/dennis/Projects/csync2/trunk/groups.c
> >> May 15 05:52:23 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/getrealfn.c): bluebox0.schafroth.dk octet-stream 36
> >> 
> >> And the command terminates premature with a "Format-error while receiving data." 
> >> 
> >> Am I the only one seeing this?  
> >> 
> >> the third string print in the "ERROR from peer" looks pretty much like random data:
> >> 
> >> May 15 05:52:24 zotac csync2: ERROR from peer(%home%/dennis/Projects/csync2/trunk/cygwin/.svn/text-base/readme_pkg.txt.svn-base): bluebox0.schafroth.dk #004l<CA><F4><A8>#013<A4>??OK (cmd_finished).
> >> 
> >> There is also an (old) issue with backup when uses prefixes, but I have a patch for that coming up. 
> > 
> > Thanks,

-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD? and LINBIT? are registered trademarks of LINBIT, Austria.

From dennis at schafroth.dk  Sun May 22 14:12:05 2011
From: dennis at schafroth.dk (Dennis Schafroth)
Date: Sun, 22 May 2011 14:12:05 +0200
Subject: [Csync2] Regression in 2.0-rc1?
In-Reply-To: <20110517112434.GA7717@barkeeper1-xen.linbit>
References: <76B02B70-914E-4B23-B738-6F294D542A8A@schafroth.dk>
	<20110516195557.GD1870@barkeeper1-xen.linbit>
	<30415343-6810-4EF5-AAF5-9DAE8377EC3B@schafroth.dk>
	<20110517112434.GA7717@barkeeper1-xen.linbit>
Message-ID: <FC65CB25-4A76-4F37-87E7-EAF28BE42D2C@schafroth.dk>


On 17/05/2011, at 13.24, Lars Ellenberg wrote:

> On Tue, May 17, 2011 at 10:16:21AM +0200, Dennis Schafroth wrote:
>> 
>> On 16/05/2011, at 21.55, Lars Ellenberg wrote:
>> 
>>> On Sun, May 15, 2011 at 03:59:47PM +0200, Dennis Schafroth wrote:
>>>> Hi 
>>>> 
>>>> I started using this yesterday, but is seeing a lot of errors: the log from the sending side looks like this:
>>> 
>>> rc1, or current git?
>> 
>> Git. Did a pull. Didnt see any comming down: 
>> 
>> commit b7490b4d2f9069e4e364c9f4824956ade4bad691
> 
>>> standalone csync2, or via inetd?
>> 
>> inetd. 
>> 
>> Since then I have tried running in standalone to debug it, but haven't been able to reproduce. So yeah, been thinking in the direction of inetd issue. 
> 
> If you can give me a short reproducer,
> I'll see to it that it gets fixed.

It's related to the new open_temp_file which tries to open temporary files in the same directory as the file it wants to check. However that idea breaks when the whole directory, where local file was located, has been deleted. So when csync_rs_check is called from csync_update_file_del sig_file will be null and the method call jump to io_error and never read the signature from the peer. 

I reverted to using a slightly modified paranoid_tmpfile but also rewrote the csync_rs_check to handle if the temp files fails. I am not sure the later is a good idea, if we wanna be paranoid.
This patch also contains the fix for backup when using prefixes.
 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: patch.diff
Type: application/octet-stream
Size: 6890 bytes
Desc: not available
URL: <http://lists.linbit.com/pipermail/csync2/attachments/20110522/d26bfdfb/attachment.obj>
-------------- next part --------------


It works for me, but please review and do your own testing :-) 

cheers, 
:-Dennis Schafroth


