[Csync2] Syncing large files takes much longer time compared to SCP or rsync

[Lars Ellenberg]

On Mon, Mar 05, 2012 at 10:50:23PM +0530, Samba wrote:
> Thanks Lars, for such a quick reply
> 
> Patches accepted.
> 
> I'm not sure if i'm competent enough, i tried fixing the issue in trust
> where csync2 does not work with different identity certificates for
> different peers, but am not successful so far :(
> 
> 
>  what csync2 is actually doing there
> >  why it takes so long
> 
> 
> I noticed that the data transfer is happening at a 512 bytes per chunk
> which results in a large number of TCP request/response cycles and thus an
> added overhead and would also slow down transfer of data.
>
> One reason could be the smaller chunks of data that the sender and receiver
> communicate with; if we know that we are dealing with larger files [by way
> of 'size' attribute from 'chktext' column, then it makes more sense to
> increase the chunk of data transfer from the present 512 bytes to some
> kilobytes [not exceeding 64K]

You can easily try: just replace the various 512 in csync_send_file
with something bigger.

But I very much doubt that will change anything.

More likely the "problem" is, that csync2 uses the "whole file" api of
librsync, so it does not pipeline sig generation, delta generation,
patch information and actual patching.

But instead first generates a temporary sig file on the whole "old" file,
then sends that sig file over, only now the peer can start with the
delta generation, again into some temp file.
Only once the delta generation is complete,
the delta patch info is sent back.

See the csync2 "rsync.c" source file.

To fix that, you'd need to switch from using rs_sig_file,
rs_loadsig_file, rs_delta_file, rs_patch_file
to the callback based variants,
and cut out the sig and delta temp files along the way.

Possibly this also needs to change the "wire protocol" of csync2
in some way. I don't know.


-- 
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com