From clifford at clifford.at  Tue Jun 13 16:31:30 2006
From: clifford at clifford.at (Clifford Wolf)
Date: Tue Jun 13 16:27:22 2006
Subject: [Csync2] Re: csync2-1.32 gets "Too many open files" error
In-Reply-To: <448DF6BE.9060101@glines.org>
References: <448DF6BE.9060101@glines.org>
Message-ID: <20060613143130.GA25312@clifford.at>

Hi,

On Mon, Jun 12, 2006 at 04:20:30PM -0700, Mark Glines wrote:
> When making large amounts of changes to a tree, csync2 dies, and needs 
> to be invoked multiple times to get the job done.
> [...]
> After a while, I get a "Too many open files" error.  The error looks 
> like this:
> [...]
> ...etc etc etc.  (there were 364 files open, when I checked just a 
> second ago.)
> 
> So, its no wonder it runs out of file handles.  Is this a design choice, 
> for some reason (rollback or something), or is it just leaking fds 
> because some subroutine didn't clean up after itself?  It seems to leave 
> both the backup-file and the original-file (now deleted) open...

I've done a test-sync with adding and removing 10000 files with an ulimit
of max 100 files (on client and server) on one of my test clusters. Afaics
the number of open files never grew over 10 (on both machines).

So maybe this is a bug in one of the backend libraries (sqlite, librsync)
which has been fixed in the versions I've been using (debian sarge):

	librsync 0.9.7-1
	libsqlite 2.8.16-1

yours,
 - clifford

PS: Mails regarding csync2 should always be sent to the csync2 mailing
list, to to me directly.

-- 
 ____   ___   ____ _  __  _     _ www.rocklinux.org
|  _ \ / _ \ / ___| |/ / | |   (_)_ __  _   ___  __
| |_) | | | | |   | ' /  | |   | | '_ \| | | \ \/ /
|  _ <| |_| | |___| . \  | |___| | | | | |_| |>  <
|_| \_\\___/ \____|_|\_\ |_____|_|_| |_|\__,_/_/\_\
 
The great thing about Object Oriented code is that it can make small,
simple problems look like large, complex ones.
 
From daniele at retaggio.net  Thu Jun 15 13:05:04 2006
From: daniele at retaggio.net (Daniele Palumbo)
Date: Thu Jun 15 13:05:27 2006
Subject: [Csync2] Can't stat large files
Message-ID: <200606151305.04347.daniele@retaggio.net>

Hi.
running csync2 1.31 (i'll try on 1.32 but...)

On files (at least) bigger than 2.6 GB, running `csync2 -xv -C config`:
"This should not happen: Can't stat [filename]"

than csync2 exit [read as: do not skip file but exit directly].
removing the files make csync2 work properly.

i have reiserfs as filesystem.

a bug? a feture? :D

bye
d.
From daniele at retaggio.net  Thu Jun 15 13:07:07 2006
From: daniele at retaggio.net (Daniele Palumbo)
Date: Thu Jun 15 13:07:16 2006
Subject: [Csync2] lockfile per configuration
Message-ID: <200606151307.07779.daniele@retaggio.net>

Hi.

I have 3 config files, and sometimes i don't want to place a global lock, but 
i prefer a lock on single configuration, so i can go on with other syncs.

is that possibile?

bye
d.
From daniele at retaggio.net  Thu Jun 15 13:16:08 2006
From: daniele at retaggio.net (Daniele Palumbo)
Date: Thu Jun 15 13:16:20 2006
Subject: [Csync2] Do not delete files
Message-ID: <200606151316.08936.daniele@retaggio.net>

Me again :D
(i promise i'll shut it down)

Can i say to csync2 to not delete files at all on one host?
so: machine A have a new file, and will send it to machine B, and C.
but suppose machine A is compromised, and a cracker delete all file of one in 
sync directory. machine A than will propagate it to machine B and C.

I want to say (maybe in config file) to B and C to NOT ALLOW it.
is that possible?

a similar thing is pretty on "updated files": if file is modified csync will 
not update it.

if you prefer: accept only new files, leave the others untouched

you can say it is an rsync job, but i prefer csync2... to "stick up all 
together".

thanks
d.

From Lars.Ellenberg at linbit.com  Thu Jun 15 13:21:41 2006
From: Lars.Ellenberg at linbit.com (Lars Ellenberg)
Date: Thu Jun 15 13:21:42 2006
Subject: [Csync2] Can't stat large files
In-Reply-To: <200606151305.04347.daniele@retaggio.net>
References: <200606151305.04347.daniele@retaggio.net>
Message-ID: <20060615112141.GB4008@soda.linbit>

/ 2006-06-15 13:05:04 +0200
\ Daniele Palumbo:
> Hi.
> running csync2 1.31 (i'll try on 1.32 but...)
> 
> On files (at least) bigger than 2.6 GB, running `csync2 -xv -C config`:
> "This should not happen: Can't stat [filename]"
> 
> than csync2 exit [read as: do not skip file but exit directly].
> removing the files make csync2 work properly.
> 
> i have reiserfs as filesystem.
> 
> a bug? a feture? :D

well.
can _you_ stat that file?

# ls -f
(i.e. opendir,readdir) shows the file.
# ls -l
(i.e. stat it, and show its time stamps and size and stuff)
gives an io-error.

maybe this happens with other filesystems too, I personally have only
seen this with reiserfs...

if thats true for your case, too, then maybe a reiserfsck is in order.
have backups handy, though...

-- 
: Lars Ellenberg                                  Tel +43-1-8178292-0  :
: LINBIT Information Technologies GmbH            Fax +43-1-8178292-82 :
: Schoenbrunner Str. 244, A-1120 Vienna/Europe   http://www.linbit.com :
From daniele at retaggio.net  Thu Jun 15 14:24:08 2006
From: daniele at retaggio.net (Daniele Palumbo)
Date: Thu Jun 15 14:24:31 2006
Subject: [Csync2] Can't stat large files
In-Reply-To: <20060615112141.GB4008@soda.linbit>
References: <200606151305.04347.daniele@retaggio.net>
	<20060615112141.GB4008@soda.linbit>
Message-ID: <200606151424.09386.daniele@retaggio.net>

On Thursday 15 June 2006 13:21, Lars Ellenberg wrote:
> well.
> can _you_ stat that file?

of course i can.

> maybe this happens with other filesystems too, I personally have only
> seen this with reiserfs...

i specified mine, to help trace problems...

> if thats true for your case, too, then maybe a reiserfsck is in order.

i'll do.
by now i can't, but i'll try to reproduce it in others system (and 
filesystem).

thanks
d.
From mathias.jansson at gmail.com  Mon Jun 19 14:45:25 2006
From: mathias.jansson at gmail.com (Mathias Jansson)
Date: Mon Jun 19 14:45:27 2006
Subject: [Csync2] Can't stat large files
In-Reply-To: <200606151305.04347.daniele@retaggio.net>
References: <200606151305.04347.daniele@retaggio.net>
Message-ID: <cea6fa6c0606190545s30e1ff72v71fadbe9ee6d067c@mail.gmail.com>

On 6/15/06, Daniele Palumbo <daniele@retaggio.net> wrote:
>
> Hi.
> running csync2 1.31 (i'll try on 1.32 but...)
>
> On files (at least) bigger than 2.6 GB, running `csync2 -xv -C config`:
> "This should not happen: Can't stat [filename]"


Reiserfs version 3.5 or lower cannot handle files bigger than 2.0Gb, check
if you are using a 3.5 fs by running:

# debugreiserfs /dev/device

Output should show what version you have on the superblock line.




-- 
Use Firefox! - http://www.mozilla.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.linbit.com/pipermail/csync2/attachments/20060619/ab7fc6d4/attachment.htm
From daniele at retaggio.net  Mon Jun 19 21:19:51 2006
From: daniele at retaggio.net (Daniele Palumbo)
Date: Mon Jun 19 21:20:36 2006
Subject: [Csync2] Can't stat large files
In-Reply-To: <cea6fa6c0606190545s30e1ff72v71fadbe9ee6d067c@mail.gmail.com>
References: <200606151305.04347.daniele@retaggio.net>
	<cea6fa6c0606190545s30e1ff72v71fadbe9ee6d067c@mail.gmail.com>
Message-ID: <4496F8D7.5090108@retaggio.net>

Mathias Jansson ha scritto:
> Reiserfs version 3.5 or lower cannot handle files bigger than 2.0Gb, 
> check if you are using a 3.5 fs by running:
> 
> # debugreiserfs /dev/device
> 
> Output should show what version you have on the superblock line.

AFAIK, 3.6.

root@data:~# debugreiserfs /dev/md5
debugreiserfs 3.6.19 (2003 www.namesys.com)


Filesystem state: consistency is not checked after last mounting

Reiserfs super block in block 16 on 0x905 of format 3.6 with standard 
journal
Count of blocks on the device: 34091904
Number of bitmaps: 1041
Blocksize: 4096
Free blocks (count of blocks - used [journal, bitmaps, data, reserved] 
blocks): 8363066
Root block: 12025699
Filesystem is NOT clean
Tree height: 5
Hash function used to sort names: "r5"
Objectid map size 62, max 972
Journal parameters:
         Device [0x0]
         Magic [0x49f960f3]
         Size 8193 blocks (including 1 for journal header) (first block 18)
         Max transaction length 1024 blocks
         Max batch size 900 blocks
         Max commit age 30
Blocks reserved by journal: 0
Fs state field: 0x0:
sb_version: 2
inode generation number: 189916
UUID: bcaaabff-9c0b-42c8-96a9-2610103bb585
LABEL:
Set flags in SB:
         ATTRIBUTES CLEAN
root@data:~#

bye
d.
From aaron.randall at visionoss.com  Tue Jun 27 14:08:56 2006
From: aaron.randall at visionoss.com (Aaron Randall)
Date: Tue Jun 27 14:08:00 2006
Subject: [Csync2] Basic Csync2 Questions
Message-ID: <44A11FD8.8020103@visionoss.com>

Hi All,

I have some basic questions that it would be great if someone could 
clear up for me.  When you create the configuration file with the lists 
of hosts and rules for how to handle conflicts etc, do you have to copy 
this to ALL hosts, and start Csync2 on ALL hosts too?

Thanks,

Aaron
From aaron.randall at visionoss.com  Thu Jun 29 10:53:39 2006
From: aaron.randall at visionoss.com (Aaron Randall)
Date: Thu Jun 29 10:52:41 2006
Subject: [Csync2] Re: Establishing SSL connection failed.
In-Reply-To: <156639090604051626m79502c86o5d91ce9463f7f24c@mail.gmail.com>
References: <156639090604051537w643a9be3r477a8d83cbce2c72@mail.gmail.com>
	<156639090604051626m79502c86o5d91ce9463f7f24c@mail.gmail.com>
Message-ID: <44A39513.8040509@visionoss.com>

can you share your solution please? I an recieving the same error :)

Eric Liang wrote:
> sorry, group. solved again. guess I was not too patient
>
> On 4/5/06, *Eric Liang* <zhenhuiliang@gmail.com 
> <mailto:zhenhuiliang@gmail.com>> wrote:
>
>     Hi, Group.
>
>     I am testing with the csync installation, install two machines
>     exactly same steps,
>     install csync2 from source, did make cert, created the key, each
>     server ran csync2 as
>     csync2 -ii
>
>     each time I do a csync2 -x, it gives me error Establishing SSL
>     connection failed.
>
>     want to troubleshoot, but don't know where to begin with.
>
>     any advices?
>
>     thanks a lot.
>
>     Eric Liang
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Csync2 mailing list
> Csync2@lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/csync2
>   
From aaron.randall at visionoss.com  Thu Jun 29 13:39:07 2006
From: aaron.randall at visionoss.com (Aaron Randall)
Date: Thu Jun 29 13:38:20 2006
Subject: [Csync2] Re: Establishing SSL connection failed.
In-Reply-To: <44A39513.8040509@visionoss.com>
References: <156639090604051537w643a9be3r477a8d83cbce2c72@mail.gmail.com>	<156639090604051626m79502c86o5d91ce9463f7f24c@mail.gmail.com>
	<44A39513.8040509@visionoss.com>
Message-ID: <44A3BBDB.8070305@visionoss.com>

ok scrap that, I just needed to do a "make cert" after compiling csync2.

cheers!

Aaron Randall wrote:
> can you share your solution please? I an recieving the same error :)
>
> Eric Liang wrote:
>> sorry, group. solved again. guess I was not too patient
>>
>> On 4/5/06, *Eric Liang* <zhenhuiliang@gmail.com 
>> <mailto:zhenhuiliang@gmail.com>> wrote:
>>
>>     Hi, Group.
>>
>>     I am testing with the csync installation, install two machines
>>     exactly same steps,
>>     install csync2 from source, did make cert, created the key, each
>>     server ran csync2 as
>>     csync2 -ii
>>
>>     each time I do a csync2 -x, it gives me error Establishing SSL
>>     connection failed.
>>
>>     want to troubleshoot, but don't know where to begin with.
>>
>>     any advices?
>>
>>     thanks a lot.
>>
>>     Eric Liang
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Csync2 mailing list
>> Csync2@lists.linbit.com
>> http://lists.linbit.com/mailman/listinfo/csync2
>>   
> _______________________________________________
> Csync2 mailing list
> Csync2@lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/csync2
>
From aaron.randall at visionoss.com  Thu Jun 29 13:42:26 2006
From: aaron.randall at visionoss.com (Aaron Randall)
Date: Thu Jun 29 13:41:24 2006
Subject: [Csync2] How to get Csync to update folders often
Message-ID: <44A3BCA2.1030608@visionoss.com>

Hi all,

This will sound like a stupid question, but I've just got csync2 working 
with two servers, but I have to use "csync2 -xv" every time I want to 
update them.  How can I get them to update the synchronized folders 
ofter (say every time a new file is created, or every few minutes or so)??

Many thanks,

Aaron
From andreas.koenig.gmwojprw at franz.ak.mind.de  Fri Jun 30 06:22:54 2006
From: andreas.koenig.gmwojprw at franz.ak.mind.de (Andreas J. Koenig)
Date: Fri Jun 30 06:24:11 2006
Subject: [Csync2] How to get Csync to update folders often
In-Reply-To: <44A3BCA2.1030608@visionoss.com> (Aaron Randall's message of
	"Thu, 29 Jun 2006 11:42:26 +0000")
References: <44A3BCA2.1030608@visionoss.com>
Message-ID: <87hd23s25d.fsf@k75.linux.bogus>

>>>>> On Thu, 29 Jun 2006 11:42:26 +0000, Aaron Randall <aaron.randall@visionoss.com> said:

  > Hi all,
  > This will sound like a stupid question, but I've just got csync2
  > working with two servers, but I have to use "csync2 -xv" every time I
  > want to update them.  How can I get them to update the synchronized
  > folders ofter (say every time a new file is created, or every few
  > minutes or so)??

"-x" does a find over the filesystem which is costly, so you do not
want to do it frequently.

If you run "-h" on every file and directory creation and deletion,
then you can run "-cu" every minute because it is very fast.

-- 
andreas
From mic at npgx.com.au  Fri Jun 30 06:31:36 2006
From: mic at npgx.com.au (Michael Mansour)
Date: Fri Jun 30 06:32:34 2006
Subject: [Csync2] How to get Csync to update folders often
In-Reply-To: <87hd23s25d.fsf@k75.linux.bogus>
References: <44A3BCA2.1030608@visionoss.com> <87hd23s25d.fsf@k75.linux.bogus>
Message-ID: <20060630043022.M93758@npgx.com.au>

Hi Andreas,

> >>>>> On Thu, 29 Jun 2006 11:42:26 +0000, Aaron Randall
<aaron.randall@visionoss.com> said:
> 
>   > Hi all,
>   > This will sound like a stupid question, but I've just got csync2
>   > working with two servers, but I have to use "csync2 -xv" every 
> time I  > want to update them.  How can I get them to update the synchronized
>   > folders ofter (say every time a new file is created, or every few
>   > minutes or so)??
> 
> "-x" does a find over the filesystem which is costly, so you do not
> want to do it frequently.
> 
> If you run "-h" on every file and directory creation and deletion,
> then you can run "-cu" every minute because it is very fast.

How do you do this?

Do you cron a "-h" periodically and then a "-cu" ?

Michael.

> -- 
> andreas
> _______________________________________________
> Csync2 mailing list
> Csync2@lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/csync2
------- End of Original Message -------

From andreas.koenig.gmwojprw at franz.ak.mind.de  Fri Jun 30 09:05:46 2006
From: andreas.koenig.gmwojprw at franz.ak.mind.de (Andreas J. Koenig)
Date: Fri Jun 30 09:09:37 2006
Subject: [Csync2] How to get Csync to update folders often
In-Reply-To: <20060630043022.M93758@npgx.com.au> (Michael Mansour's message of
	"Fri, 30 Jun 2006 14:31:36 +1000")
References: <44A3BCA2.1030608@visionoss.com> <87hd23s25d.fsf@k75.linux.bogus>
	<20060630043022.M93758@npgx.com.au>
Message-ID: <877j2zrulx.fsf@k75.linux.bogus>

>>>>> On Fri, 30 Jun 2006 14:31:36 +1000, "Michael Mansour" <mic@npgx.com.au> said:

  > Hi Andreas,
 >> >>>>> On Thu, 29 Jun 2006 11:42:26 +0000, Aaron Randall
  > <aaron.randall@visionoss.com> said:
 >> 
 >> > Hi all,
 >> > This will sound like a stupid question, but I've just got csync2
 >> > working with two servers, but I have to use "csync2 -xv" every 
 >> time I  > want to update them.  How can I get them to update the synchronized
 >> > folders ofter (say every time a new file is created, or every few
 >> > minutes or so)??
 >> 
 >> "-x" does a find over the filesystem which is costly, so you do not
 >> want to do it frequently.
 >> 
 >> If you run "-h" on every file and directory creation and deletion,
 >> then you can run "-cu" every minute because it is very fast.

  > How do you do this?

  > Do you cron a "-h" periodically and then a "-cu" ?

No, I have hooks in most programs that write to the disk or remove
from the disk. These hooks call 'csync2 -h somepath' immediately after
writing or deleting. Then I run 'csync2 -cu' every minute. Every 30
minutes I run an additional "csync2 -cr /" so that all file events get
caught that did something on the disk bypassing the hooks.

-- 
andreas
