[Csync2] Syncing large files takes much longer time compared to SCP or rsync

[Lars Ellenberg]

On Mon, Mar 05, 2012 at 09:55:17PM +0530, Samba wrote:
> Hi,
> 
> 
> I'm using CSYNC2 2.0-rc1 from the git stream on CENT OS 5.6 64 bit...
> 
> CSync2 is consuming lot of cpu cycles and taking a lot of time [roughly,
> around 10 times] compared to SCP or RSYNC; are there any special options to
> make csync2 work like scp/rsync style copy for large files?
> 
> If we clearly know that certain files will only be added to or removed from
> a directory and no modifications happen to any of those large files, then
> would it not be beneficial to just plainly copy instead of finding
> differences, infact there won't be any differences, just 1 or 0 for
> existence or  absence of file.

Patches accepted.

> I tried initializing the csync2 metadata database with csync2 -crI and then
> did an csync -x, but it is taking a lot of time even on a LAN when compared
> to other copy utilities like rsync or scp.
> 
> One may ask why use csync2 and why not go with scp or rsync for such use
> cases; the only answer i can give is that those tools are no match for the
> flexibility and power of csync2 and i would want to address this use case
> also along with the other common scenarios that csync2 excels in.
> 
> Please let me if there is any way where i can force csync2 to straightway
> sync the file like plain rsync and do not try to find diffs and thus
> improve speed and reduce resource consumption?
> 
> Thanks and Regards,
> Samba
> *
> *
> 
> PS:
> 
> Here are the results for times taken for copying a nearly 1 GB file between
> two hosts in the same LAN:
> 
> *SCP *
> 
> [root at master~]# time scp /opt/test/installer.zip root at slave:/opt/test/
> 
> real    0m35.041s
> 
> user    0m11.590s
> 
> sys     0m8.933s
> 
> 
> *RSYNC*
> 
> [root at master~]# time rsync /opt/test/installer.zip root at slave:/opt/test/
> 
> real    0m32.724s
> 
> user    0m10.987s
> 
> sys     0m4.576s
> 
> 
> *CSYNC2*
> 
> [root at master~]# time csync2 -x

why not: time csync2 -x /opt/test/installer.zip

> real    5m53.171s
> 
> user    4m19.255s
> 
> sys     0m31.349s

I suggest you figure out what csync2 is actually doing there,
why it takes so long.

strace?
add sufficient numbers of "-v"?

Do you have thousands of files,
and are unaware that -x implies -c,
which implies a full recursive stat run?


-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.