<blockquote class="webkit-indent-blockquote" style="margin:0 0 0 40px;border:none;padding:0px"></blockquote>Thanks Lars, for such a quick reply<br><br><blockquote class="webkit-indent-blockquote" style="margin:0 0 0 40px;border:none;padding:0px">
<blockquote class="gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Patches accepted.</blockquote>
<blockquote class="webkit-indent-blockquote" style="margin:0 0 0 40px;border:none;padding:0px">I'm not sure if i'm competent enough, i tried fixing the issue in trust where csync2 does not work with different identity certificates for different peers, but am not successful so far :(</blockquote>
<br><blockquote class="gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> what csync2 is actually doing there<br>
why it takes so long</blockquote><br>I noticed that the data transfer is happening at a 512 bytes per chunk which results in a large number of TCP request/response cycles and thus an added overhead and would also slow down transfer of data.<br>
<br>One reason could be the smaller chunks of data that the sender and receiver communicate with; if we know that we are dealing with larger files [by way of 'size' attribute from 'chktext' column, then it makes more sense to increase the chunk of data transfer from the present 512 bytes to some kilobytes [not exceeding 64K]<br>
<br></blockquote>Regards,<br>Samba<br><br>---------------------------------------------------------------------------------------------------------------------------------------------------------------------<br><blockquote class="webkit-indent-blockquote" style="margin:0 0 0 40px;border:none;padding:0px">
<br><div class="gmail_quote">On Mon, Mar 5, 2012 at 10:05 PM, Lars Ellenberg <span dir="ltr"><<a href="mailto:lars.ellenberg@linbit.com">lars.ellenberg@linbit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On Mon, Mar 05, 2012 at 09:55:17PM +0530, Samba wrote:<br>
> Hi,<br>
><br>
><br>
> I'm using CSYNC2 2.0-rc1 from the git stream on CENT OS 5.6 64 bit...<br>
><br>
> CSync2 is consuming lot of cpu cycles and taking a lot of time [roughly,<br>
> around 10 times] compared to SCP or RSYNC; are there any special options to<br>
> make csync2 work like scp/rsync style copy for large files?<br>
><br>
> If we clearly know that certain files will only be added to or removed from<br>
> a directory and no modifications happen to any of those large files, then<br>
> would it not be beneficial to just plainly copy instead of finding<br>
> differences, infact there won't be any differences, just 1 or 0 for<br>
> existence or absence of file.<br>
<br>
</div>Patches accepted.<br>
<div class="im"><br>
> I tried initializing the csync2 metadata database with csync2 -crI and then<br>
> did an csync -x, but it is taking a lot of time even on a LAN when compared<br>
> to other copy utilities like rsync or scp.<br>
><br>
> One may ask why use csync2 and why not go with scp or rsync for such use<br>
> cases; the only answer i can give is that those tools are no match for the<br>
> flexibility and power of csync2 and i would want to address this use case<br>
> also along with the other common scenarios that csync2 excels in.<br>
><br>
> Please let me if there is any way where i can force csync2 to straightway<br>
> sync the file like plain rsync and do not try to find diffs and thus<br>
> improve speed and reduce resource consumption?<br>
><br>
> Thanks and Regards,<br>
> Samba<br>
</div>> *<br>
<div class="im">> *<br>
><br>
> PS:<br>
><br>
> Here are the results for times taken for copying a nearly 1 GB file between<br>
> two hosts in the same LAN:<br>
><br>
</div>> *SCP *<br>
<div class="im">><br>
> [root@master~]# time scp /opt/test/installer.zip root@slave:/opt/test/<br>
><br>
> real 0m35.041s<br>
><br>
> user 0m11.590s<br>
><br>
> sys 0m8.933s<br>
><br>
><br>
</div>> *RSYNC*<br>
<div class="im">><br>
> [root@master~]# time rsync /opt/test/installer.zip root@slave:/opt/test/<br>
><br>
> real 0m32.724s<br>
><br>
> user 0m10.987s<br>
><br>
> sys 0m4.576s<br>
><br>
><br>
</div>> *CSYNC2*<br>
<div class="im">><br>
> [root@master~]# time csync2 -x<br>
<br>
</div>why not: time csync2 -x /opt/test/installer.zip<br>
<div class="im"><br>
> real 5m53.171s<br>
><br>
> user 4m19.255s<br>
><br>
> sys 0m31.349s<br>
<br>
</div>I suggest you figure out what csync2 is actually doing there,<br>
why it takes so long.<br>
<br>
strace?<br>
add sufficient numbers of "-v"?<br>
<br>
Do you have thousands of files,<br>
and are unaware that -x implies -c,<br>
which implies a full recursive stat run?<br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
--<br>
: Lars Ellenberg<br>
: LINBIT | Your Way to High Availability<br>
: DRBD/HA support and consulting <a href="http://www.linbit.com" target="_blank">http://www.linbit.com</a><br>
<br>
DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.<br>
_______________________________________________<br>
Csync2 mailing list<br>
<a href="mailto:Csync2@lists.linbit.com">Csync2@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/csync2" target="_blank">http://lists.linbit.com/mailman/listinfo/csync2</a><br>
</font></span></blockquote></div><br><blockquote class="gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
what csync2 is actually doing there</blockquote><blockquote class="gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
why it takes so long</blockquote><br>I noticed that the data transfer is happening at a 512 bytes per chunk which results in a large number of TCP request/response cycles and thus an added overhead and would also slow down transfer of data.<br>
<br>One reason could be the smaller chunks of data that the sender and receiver communicate with; if we know that we are dealing with larger files [by way of 'size' attribute from 'chktext' column, then it makes more sense to increase the chunk of data transfer from the present 512 bytes to some kilobytes [not exceeding 64K]<br>
<br><br><br><div class="gmail_quote">On Mon, Mar 5, 2012 at 10:05 PM, Lars Ellenberg <span dir="ltr"><<a href="mailto:lars.ellenberg@linbit.com">lars.ellenberg@linbit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On Mon, Mar 05, 2012 at 09:55:17PM +0530, Samba wrote:<br>
> Hi,<br>
><br>
><br>
> I'm using CSYNC2 2.0-rc1 from the git stream on CENT OS 5.6 64 bit...<br>
><br>
> CSync2 is consuming lot of cpu cycles and taking a lot of time [roughly,<br>
> around 10 times] compared to SCP or RSYNC; are there any special options to<br>
> make csync2 work like scp/rsync style copy for large files?<br>
><br>
> If we clearly know that certain files will only be added to or removed from<br>
> a directory and no modifications happen to any of those large files, then<br>
> would it not be beneficial to just plainly copy instead of finding<br>
> differences, infact there won't be any differences, just 1 or 0 for<br>
> existence or absence of file.<br>
<br>
</div>Patches accepted.<br>
<div class="im"><br>
> I tried initializing the csync2 metadata database with csync2 -crI and then<br>
> did an csync -x, but it is taking a lot of time even on a LAN when compared<br>
> to other copy utilities like rsync or scp.<br>
><br>
> One may ask why use csync2 and why not go with scp or rsync for such use<br>
> cases; the only answer i can give is that those tools are no match for the<br>
> flexibility and power of csync2 and i would want to address this use case<br>
> also along with the other common scenarios that csync2 excels in.<br>
><br>
> Please let me if there is any way where i can force csync2 to straightway<br>
> sync the file like plain rsync and do not try to find diffs and thus<br>
> improve speed and reduce resource consumption?<br>
><br>
> Thanks and Regards,<br>
> Samba<br>
</div>> *<br>
<div class="im">> *<br>
><br>
> PS:<br>
><br>
> Here are the results for times taken for copying a nearly 1 GB file between<br>
> two hosts in the same LAN:<br>
><br>
</div>> *SCP *<br>
<div class="im">><br>
> [root@master~]# time scp /opt/test/installer.zip root@slave:/opt/test/<br>
><br>
> real 0m35.041s<br>
><br>
> user 0m11.590s<br>
><br>
> sys 0m8.933s<br>
><br>
><br>
</div>> *RSYNC*<br>
<div class="im">><br>
> [root@master~]# time rsync /opt/test/installer.zip root@slave:/opt/test/<br>
><br>
> real 0m32.724s<br>
><br>
> user 0m10.987s<br>
><br>
> sys 0m4.576s<br>
><br>
><br>
</div>> *CSYNC2*<br>
<div class="im">><br>
> [root@master~]# time csync2 -x<br>
<br>
</div>why not: time csync2 -x /opt/test/installer.zip<br>
<div class="im"><br>
> real 5m53.171s<br>
><br>
> user 4m19.255s<br>
><br>
> sys 0m31.349s<br>
<br>
</div>I suggest you figure out what csync2 is actually doing there,<br>
why it takes so long.<br>
<br>
strace?<br>
add sufficient numbers of "-v"?<br>
<br>
Do you have thousands of files,<br>
and are unaware that -x implies -c,<br>
which implies a full recursive stat run?<br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
--<br>
: Lars Ellenberg<br>
: LINBIT | Your Way to High Availability<br>
: DRBD/HA support and consulting <a href="http://www.linbit.com" target="_blank">http://www.linbit.com</a><br>
<br>
DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.<br>
_______________________________________________<br>
Csync2 mailing list<br>
<a href="mailto:Csync2@lists.linbit.com">Csync2@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/csync2" target="_blank">http://lists.linbit.com/mailman/listinfo/csync2</a><br>
</font></span></blockquote></div><br>I noticed that the data transfer is happening at a 512 bytes per chunk which results in a large number of TCP request/response cycles and thus an added overhead and would also slow down transfer of data.<br>
One reason could be the smaller chunks of data that the sender and receiver communicate with; if we know that we are dealing with larger files [by way of 'size' attribute from 'chktext' column, then it makes more sense to increase the chunk of data transfer from the present 512 bytes to some kilobytes [not exceeding 64K]<br>
<br><br><br><div class="gmail_quote">On Mon, Mar 5, 2012 at 10:05 PM, Lars Ellenberg <span dir="ltr"><<a href="mailto:lars.ellenberg@linbit.com">lars.ellenberg@linbit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On Mon, Mar 05, 2012 at 09:55:17PM +0530, Samba wrote:<br>
> Hi,<br>
><br>
><br>
> I'm using CSYNC2 2.0-rc1 from the git stream on CENT OS 5.6 64 bit...<br>
><br>
> CSync2 is consuming lot of cpu cycles and taking a lot of time [roughly,<br>
> around 10 times] compared to SCP or RSYNC; are there any special options to<br>
> make csync2 work like scp/rsync style copy for large files?<br>
><br>
> If we clearly know that certain files will only be added to or removed from<br>
> a directory and no modifications happen to any of those large files, then<br>
> would it not be beneficial to just plainly copy instead of finding<br>
> differences, infact there won't be any differences, just 1 or 0 for<br>
> existence or absence of file.<br>
<br>
</div>Patches accepted.<br>
<div class="im"><br>
> I tried initializing the csync2 metadata database with csync2 -crI and then<br>
> did an csync -x, but it is taking a lot of time even on a LAN when compared<br>
> to other copy utilities like rsync or scp.<br>
><br>
> One may ask why use csync2 and why not go with scp or rsync for such use<br>
> cases; the only answer i can give is that those tools are no match for the<br>
> flexibility and power of csync2 and i would want to address this use case<br>
> also along with the other common scenarios that csync2 excels in.<br>
><br>
> Please let me if there is any way where i can force csync2 to straightway<br>
> sync the file like plain rsync and do not try to find diffs and thus<br>
> improve speed and reduce resource consumption?<br>
><br>
> Thanks and Regards,<br>
> Samba<br>
</div>> *<br>
<div class="im">> *<br>
><br>
> PS:<br>
><br>
> Here are the results for times taken for copying a nearly 1 GB file between<br>
> two hosts in the same LAN:<br>
><br>
</div>> *SCP *<br>
<div class="im">><br>
> [root@master~]# time scp /opt/test/installer.zip root@slave:/opt/test/<br>
><br>
> real 0m35.041s<br>
><br>
> user 0m11.590s<br>
><br>
> sys 0m8.933s<br>
><br>
><br>
</div>> *RSYNC*<br>
<div class="im">><br>
> [root@master~]# time rsync /opt/test/installer.zip root@slave:/opt/test/<br>
><br>
> real 0m32.724s<br>
><br>
> user 0m10.987s<br>
><br>
> sys 0m4.576s<br>
><br>
><br>
</div>> *CSYNC2*<br>
<div class="im">><br>
> [root@master~]# time csync2 -x<br>
<br>
</div>why not: time csync2 -x /opt/test/installer.zip<br>
<div class="im"><br>
> real 5m53.171s<br>
><br>
> user 4m19.255s<br>
><br>
> sys 0m31.349s<br>
<br>
</div>I suggest you figure out what csync2 is actually doing there,<br>
why it takes so long.<br>
<br>
strace?<br>
add sufficient numbers of "-v"?<br>
<br>
Do you have thousands of files,<br>
and are unaware that -x implies -c,<br>
which implies a full recursive stat run?<br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
--<br>
: Lars Ellenberg<br>
: LINBIT | Your Way to High Availability<br>
: DRBD/HA support and consulting <a href="http://www.linbit.com" target="_blank">http://www.linbit.com</a><br>
<br>
DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.<br>
_______________________________________________<br>
Csync2 mailing list<br>
<a href="mailto:Csync2@lists.linbit.com">Csync2@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/csync2" target="_blank">http://lists.linbit.com/mailman/listinfo/csync2</a><br>
</font></span></blockquote></div><br>One reason could be the smaller chunks of data that the sender and receiver communicate with; if we know that we are dealing with larger files [by way of 'size' attribute from 'chktext' column, then it makes more sense to increase the chunk of data transfer from the present 512 bytes to some kilobytes [not exceeding 64K]</blockquote>
<br><br><br><div class="gmail_quote">On Mon, Mar 5, 2012 at 10:05 PM, Lars Ellenberg <span dir="ltr"><<a href="mailto:lars.ellenberg@linbit.com">lars.ellenberg@linbit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On Mon, Mar 05, 2012 at 09:55:17PM +0530, Samba wrote:<br>
> Hi,<br>
><br>
><br>
> I'm using CSYNC2 2.0-rc1 from the git stream on CENT OS 5.6 64 bit...<br>
><br>
> CSync2 is consuming lot of cpu cycles and taking a lot of time [roughly,<br>
> around 10 times] compared to SCP or RSYNC; are there any special options to<br>
> make csync2 work like scp/rsync style copy for large files?<br>
><br>
> If we clearly know that certain files will only be added to or removed from<br>
> a directory and no modifications happen to any of those large files, then<br>
> would it not be beneficial to just plainly copy instead of finding<br>
> differences, infact there won't be any differences, just 1 or 0 for<br>
> existence or absence of file.<br>
<br>
</div>Patches accepted.<br>
<div class="im"><br>
> I tried initializing the csync2 metadata database with csync2 -crI and then<br>
> did an csync -x, but it is taking a lot of time even on a LAN when compared<br>
> to other copy utilities like rsync or scp.<br>
><br>
> One may ask why use csync2 and why not go with scp or rsync for such use<br>
> cases; the only answer i can give is that those tools are no match for the<br>
> flexibility and power of csync2 and i would want to address this use case<br>
> also along with the other common scenarios that csync2 excels in.<br>
><br>
> Please let me if there is any way where i can force csync2 to straightway<br>
> sync the file like plain rsync and do not try to find diffs and thus<br>
> improve speed and reduce resource consumption?<br>
><br>
> Thanks and Regards,<br>
> Samba<br>
</div>> *<br>
<div class="im">> *<br>
><br>
> PS:<br>
><br>
> Here are the results for times taken for copying a nearly 1 GB file between<br>
> two hosts in the same LAN:<br>
><br>
</div>> *SCP *<br>
<div class="im">><br>
> [root@master~]# time scp /opt/test/installer.zip root@slave:/opt/test/<br>
><br>
> real 0m35.041s<br>
><br>
> user 0m11.590s<br>
><br>
> sys 0m8.933s<br>
><br>
><br>
</div>> *RSYNC*<br>
<div class="im">><br>
> [root@master~]# time rsync /opt/test/installer.zip root@slave:/opt/test/<br>
><br>
> real 0m32.724s<br>
><br>
> user 0m10.987s<br>
><br>
> sys 0m4.576s<br>
><br>
><br>
</div>> *CSYNC2*<br>
<div class="im">><br>
> [root@master~]# time csync2 -x<br>
<br>
</div>why not: time csync2 -x /opt/test/installer.zip<br>
<div class="im"><br>
> real 5m53.171s<br>
><br>
> user 4m19.255s<br>
><br>
> sys 0m31.349s<br>
<br>
</div>I suggest you figure out what csync2 is actually doing there,<br>
why it takes so long.<br>
<br>
strace?<br>
add sufficient numbers of "-v"?<br>
<br>
Do you have thousands of files,<br>
and are unaware that -x implies -c,<br>
which implies a full recursive stat run?<br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
--<br>
: Lars Ellenberg<br>
: LINBIT | Your Way to High Availability<br>
: DRBD/HA support and consulting <a href="http://www.linbit.com" target="_blank">http://www.linbit.com</a><br>
<br>
DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.<br>
_______________________________________________<br>
Csync2 mailing list<br>
<a href="mailto:Csync2@lists.linbit.com">Csync2@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/csync2" target="_blank">http://lists.linbit.com/mailman/listinfo/csync2</a><br>
</font></span></blockquote></div><br>