Mark,<br><br>syncer rate does not define the overall drbd performance between the two nodes, but rather it designates a limit for re-syncing, to assure normal system performance while it is happening in the backgrounnd (so your re-sync processes dont eat up all your bandwidth between your drbd nodes, and normal drbd usage can happen). <br>
<br>It has nothing to do with the working performance of a single drbd device pair see more here: <a href="http://www.drbd.org/users-guide/s-configure-syncer-rate.html">http://www.drbd.org/users-guide/s-configure-syncer-rate.html</a><br>
<br>Originally it was set to be 80M in this case, but probably it should be even lower, since despite the iperf results, I never saw a drbd go over the 117MB/s (single gigabit link hitting performance wall), so it probably should be as low as 35MB/s<br>
<br>As I wrote before, the two boxes have three gigabit links dedicated for DRBD, these are bonded using the balance-rr mode, and with arp ip monitoring (basically I can unplug any of the cables between the nodes, in any order, and plug them back in, as long there is a single link, the connection is uninterrupted, also depending how many gigabit links are allive the bandwidth scales up with every additional connection, pretty swift actually)<br>
<br>What I would like to see is higher write rates. I know how the different bondings work, I also know balance-rr is the only where a single connection cab scale beyond the capacity of a single card, and it clearly happens when I benchamrk it with iperf, but never saw it in DRBD.<br>
<br>Now, since this is a Xen Dom0, I have been able to do more testing in the DomU itself (half of the testing was done before I wrote to the mailing list)<br><br>In the paravirtualized DomU, the drbd devices are inported as xvdb to xvdf, and they were used:<br>
<br>1) as phisical volumes for the LVM in the DomU itseld, setting up the logical volumes using striping in LVM<br><br>2) as part of a raid0 stripe<br><br>3) raid0 stripe in above as a phisical volume for LVM<br><br>There is no signofocant difference in either mode.<br>
<br>I have even tested a RAID_0 stripe over the drbd volumes in the Xen Dom0, same results.<br><br>I also know about the LVM default block device performance issues, and "blockdev --setra" is used as a workaround on all levels<br>
<br>Whenever I disconnect drbd, performance is as expected (close to the raid10 performance measured)<br><br>The reason this is so annoying, is because on both the DRBD wiki and in the mail list there are hints that even over a dual gigabit link in ballance-rr performance is much better, see them yourself:<br>
<br><a href="http://www.drbd.org/home/wiki/?tx_drwiki_pi1[keyword]=performance">http://www.drbd.org/home/wiki/?tx_drwiki_pi1[keyword]=performance</a><br><br><a href="http://www.nabble.com/DRBD-Performance-td18745802.html">http://www.nabble.com/DRBD-Performance-td18745802.html</a><br>
<br><a href="http://lists.linbit.com/pipermail/drbd-user/2008-July/009893.html">http://lists.linbit.com/pipermail/drbd-user/2008-July/009893.html</a><br><br><br>Also if it not clear, this is a nested LVM (with the DRBD's being the phisical volumes for LVM in the paravirtualized xen instance, DRBD is running in Dom0):<br>
<br> DRBD<-----| |<-----xvdb (PV)|<br>Xen Dom0 6XHDD<-----RAID10<-----LVM<-----DRBD<-----|XenDomU|<-----xvdc (PV)|<-----LVM<----file ystems<br>
DRBD<-----| |<-----xvdd|(PV)|<br><br>As a side note, I am a seasoned sysadmin of fifteen years and use linux practically for everything for the last ten years, work with it at least twelve hours a day (mostly more than that, I am lucky to do for living what I love, and work and fun are the same)<br>
<br>So, anybody knows how those magical numbers were achived under those links?<br><br>z<br><br><div class="gmail_quote">On Thu, Jul 30, 2009 at 9:18 AM, Mark Watts <span dir="ltr"><<a href="mailto:m.watts@eris.qinetiq.com">m.watts@eris.qinetiq.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="im">On Thu, 2009-07-30 at 03:57 -0400, Zoltan Patay wrote:<br>
> using "dd if=/dev/zero of=/dev/drbd26 bs=10M count=100" I get:<br>
><br>
> drbd connected<br>
> 1048576000 bytes (1.0 GB) copied, 13.6526 seconds, 76.8 MB/s<br>
> 1048576000 bytes (1.0 GB) copied, 13.4238 seconds, 78.1 MB/s<br>
> 1048576000 bytes (1.0 GB) copied, 13.2448 seconds, 79.2 MB/s<br>
><br>
> drbd disconnected<br>
> 1048576000 bytes (1.0 GB) copied, 4.04754 seconds, 259 MB/s<br>
> 1048576000 bytes (1.0 GB) copied, 4.06758 seconds, 258 MB/s<br>
> 1048576000 bytes (1.0 GB) copied, 4.06758 seconds, 258 MB/s<br>
><br>
> The three (intel) gigabit PCIe cards are bonded with balance-rr, and<br>
> iperf gives me:<br>
><br>
> iperf 0.0-10.0 sec 2.52 GBytes 2.16 Gbits/sec (276.48MB/s)<br>
><br>
> So clearly there is enough speed for both on the network and in the<br>
> backend to support higher speeds. The boxes are with cross-over<br>
> back-to-back no-switch.<br>
><br>
> version: 8.3.0 (api:88/proto:86-89)<br>
> GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by<br>
> phil@fat-tyre, 2008-12-18 15:26:13<br>
><br>
> global { usage-count yes; }<br>
> common { syncer { rate 650M; } }<br>
<br>
</div>Try actually setting this to a sensible value for 3 x 1Gbit links.<br>
eg: 300M<br>
<div class="im"><br>
> resource OpenVZ_C1C2_B_LVM5 {<br>
> protocol C;<br>
> startup {degr-wfc-timeout 120;}<br>
> disk {on-io-error<br>
> detach;no-disk-flushes;no-md-flushes;no-disk-drain;no-disk-barrier;}<br>
> net {<br>
> cram-hmac-alg sha1;<br>
> shared-secret "OpenVZ_C1C2_B";<br>
> allow-two-primaries;<br>
> after-sb-0pri discard-zero-changes;<br>
> after-sb-1pri discard-secondary;<br>
> after-sb-2pri disconnect;<br>
> rr-conflict disconnect;<br>
> timeout 300;<br>
> connect-int 10;<br>
> ping-int 10;<br>
> max-buffers 2048;<br>
> max-epoch-size 2048;<br>
> }<br>
> syncer {rate 650M;al-extents 257;verify-alg crc32c;}<br>
<br>
</div>And here too.<br>
<br>
<br>
Mark.<br>
<font color="#888888">--<br>
Mark Watts BSc RHCE MBCS<br>
Senior Systems Engineer, Managed Services Manpower<br>
<a href="http://www.QinetiQ.com" target="_blank">www.QinetiQ.com</a><br>
QinetiQ - Delivering customer-focused solutions<br>
GPG Key: <a href="http://www.linux-corner.info/mwatts.gpg" target="_blank">http://www.linux-corner.info/mwatts.gpg</a><br>
</font></blockquote></div><br>