Hi all,<br><br>I'm currently playing with DRBD (8.2.7) on 20Gb/s Infiniband, and it seems that I'm running at the sync rate as the limiting speed factor. The local storage on both nodes is identical (SAS array), and has been benchmarked at about 650MB/s (or higher, depending on benchmark) to native disk, and about 550MB/s when writing to it through a disconnected DRBD device. The network link for DRBD is Infiniband as well (IPoIB), which has been benchmarked with netperf at ~800MB/s.<br>
<br>The fastest speed that I'm able to get from the DRBD sync with this configuration is ~340MB/s, which limits the speed from my initiator to that as well. Interestingly, I was also able to benchmark DRBD sync speed over 10Gbe, which despite my repeated attempts to tweak drbd.conf, mtu, and tcp kernel parameters, has produced the same speed as the aformentioned 340MB/s over IPoIB.<br>
<br>Here's the drbd.conf:<br><br>global {<br> usage-count yes;<br>}<br><br>common {<br> syncer { <br> rate 900M;<br> }<br>}<br><br>resource drbd0 {<br><br> protocol C;<br><br> handlers {<br> }<br><br>
startup {<br> degr-wfc-timeout 30;<br> }<br><br> disk {<br> on-io-error detach;<br> fencing dont-care;<br> no-disk-flushes;<br> no-md-flushes;<br> no-disk-drain;<br> no-disk-barrier;<br> }<br><br>
net {<br> ko-count 2;<br> after-sb-1pri discard-secondary;<br> sndbuf-size 1M;<br> }<br><br> on srpt1 {<br> device /dev/drbd0;<br> disk /dev/sdb;<br> address <a href="http://10.0.0.2:7789">10.0.0.2:7789</a>;<br>
flexible-meta-disk internal;<br> }<br><br> on srpt2 {<br> device /dev/drbd0;<br> disk /dev/sdb;<br> address <a href="http://10.0.0.3:7789">10.0.0.3:7789</a>;<br> flexible-meta-disk internal;<br>
}<br>}<br><br>Any advice/thoughts would be highly appreciated; thanks!<br><br>-Gennadiy<br>