[Csync2] what is for and not for

[Lars Ellenberg]

On Wed, May 23, 2012 at 04:48:22PM +0200, Thomas -Balu- Walter wrote:
> Hi everyone,
> 
> Am 23.05.12 15:39, schrieb Łukasz Wąsikowski:
> > W dniu 2012-05-23 15:27, lejeczek pisze:
> >
> >> I wonder, is csync2 right solution for data trees comprising of several
> >> thousands of files?
> >> if not do you know when, at what limits, one may begin to misuse csync2?
> >
> > $ sqlite /var/lib/csync2/`hostname`.db
> > SQLite version 2.8.16
> > Enter ".help" for instructions
> > sqlite>  select count(*) from file;
> > 57882
> > sqlite>
> >
> > Works like a charm. Problems (too long synchronization times for my
> > needs) starts above 200k files.
> 
> sqlite> select count(*) from file;
> 266797
> 
> We sync a mix of files (html, php, images, flash, movies) to two
> destination hosts about every minute. If one sync takes longer than
> this minute, the next one will just skip.
> 
> # time csync2 -x
> 
> real	0m12.914s
> user	0m7.016s
> sys	0m5.168s
> 
> The CMS helps by marking files as dirty after it changed them.

csync2 -x is equivalent to "csync2 -cr / && csync2 -u".

If your CMS helps by marking files as dirty, you should
only use the "-cr /" part (full recursive scan) occasionally.

There is csync2 -u, which will only update (sync) those that
have already be marked dirty in the database (and not recursively
scan/stat the full set).

There is the "hints" table, which can be populated via e.g.
the contributed inotify daemon, or explicitly by "csync2 -h <file...>".
The hints table is used by a "csync2 -c", without any other parameters.


If you think you need frequent "-cr /", but all that stat()ing takes too
long, then adding more RAM may help, reducing VM presure, and/or tuning
vm.vfs_cache_pressure, so the inode and dentry caches need not be
re-populated from disk every time.

There are installations with several ten nodes and some million files.

It all depends on what you really want to do,
what your requirements and expectations are,
and if you can find a better tool for the job.


-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com