<div dir="ltr">Hi Digimer,<div><br></div><div>Following is my qperf test between 2 nodes. TCP_BW test is good enough but something weird on UDP_BW test. What do you think? <a href="http://192.168.107.0/24">192.168.107.0/24</a> is on eth0 and <a href="http://10.10.130.0/24">10.10.130.0/24</a> is on eth1. Currently DRBD using eth1.</div><div><br></div><div><div>[root@db-2 ~]# qperf -t 60 --use_bits_per_sec 192.168.107.13 tcp_bw</div><div>tcp_bw:</div><div> bw = 8.81 Gb/sec</div><div><div><div>[root@db-2 ~]# qperf -t 60 --use_bits_per_sec 192.168.107.13 udp_bw</div><div>udp_bw:</div><div> send_bw = 13.6 Gb/sec</div><div> recv_bw = 5.82 Gb/sec</div></div></div><div>[root@db-2 ~]# qperf -t 60 --use_bits_per_sec 10.10.130.9 tcp_bw<br></div><div>tcp_bw:</div><div> bw = 8.47 Gb/sec</div><div>[root@db-2 ~]# qperf -t 60 --use_bits_per_sec 10.10.130.9 udp_bw</div><div>udp_bw:</div><div> send_bw = 13.2 Gb/sec</div><div> recv_bw = 79.6 Mb/sec</div><div><br></div></div><div>Best regards,</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Feb 7, 2017 at 2:47 PM, Digimer <span dir="ltr"><<a href="mailto:lists@alteeve.ca" target="_blank">lists@alteeve.ca</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 06/02/17 02:08 PM, Lazuardi Nasution wrote:<br>
> Hi,<br>
><br>
> I'm new with DRBD. I'm trying to setup dual primary nodes (VMs with<br>
> virtio-net of bonded of dual 10GbE links) with following resource config.<br>
><br>
> resource db {<br>
> on db-1 {<br>
> volume 0 {<br>
> device /dev/drbd0 minor 0;<br>
> disk /dev/vdc1;<br>
> meta-disk internal;<br>
> }<br>
> address ipv4 <a href="http://10.10.130.9:7788" rel="noreferrer" target="_blank">10.10.130.9:7788</a><br>
</span>> <<a href="http://10.10.130.9:7788" rel="noreferrer" target="_blank">http://10.10.130.9:7788</a>>;<br>
<span class="">> }<br>
> on db-2 {<br>
> volume 0 {<br>
> device /dev/drbd0 minor 0;<br>
> disk /dev/vdc1;<br>
> meta-disk internal;<br>
> }<br>
> address ipv4 <a href="http://10.10.130.10:7788" rel="noreferrer" target="_blank">10.10.130.10:7788</a><br>
</span>> <<a href="http://10.10.130.10:7788" rel="noreferrer" target="_blank">http://10.10.130.10:7788</a>>;<br>
<div><div class="h5">> }<br>
> options {<br>
> on-no-data-accessible io-error;<br>
> }<br>
> net {<br>
> protocol C;<br>
> allow-two-primaries yes;<br>
> after-sb-0pri discard-zero-changes;<br>
> after-sb-1pri discard-secondary;<br>
> after-sb-2pri disconnect;<br>
> sndbuf-size 1M;<br>
> rcvbuf-size 2M;<br>
> max-buffers 131072;<br>
> max-epoch-size 20000;<br>
> cram-hmac-alg sha1;<br>
> shared-secret db;<br>
> }<br>
> disk {<br>
> on-io-error detach;<br>
> disk-flushes no;<br>
> disk-barrier no;<br>
> resync-rate 1G;<br>
> al-extents 257;<br>
> c-plan-ahead 8;<br>
> c-fill-target 25M;<br>
> c-max-rate 1G;<br>
> c-min-rate 100M;<br>
> }<br>
> startup {<br>
> wfc-timeout 30;<br>
> outdated-wfc-timeout 20;<br>
> degr-wfc-timeout 30;<br>
> become-primary-on both;<br>
> }<br>
> }<br>
><br>
><br>
> I have tried to change some of variables, but no matter I have done, the<br>
> performance is just around 70MB/s like dd result below.<br>
><br>
> [root@db-1 ~]# dd if=/dev/zero of=/dev/drbd0 bs=4194304 count=1000<br>
> 1000+0 records in<br>
> 1000+0 records out<br>
> <a href="tel:4194304000" value="+14194304000">4194304000</a> bytes (4.2 GB) copied, 61.7918 s, 67.9 MB/s<br>
><br>
><br>
> The same test to the backing storage can give aroud 700MB/s performance.<br>
> What should I do with this case?<br>
><br>
> Best regards,<br>
<br>
</div></div>There are a few things here;<br>
<br>
1. DRBD has sensible defaults. Start by dramatically simplifying your<br>
config to only the specifics you need. Tune later after you have a<br>
baseline. You'll find, I suspect, your tuning won't change much, or<br>
often actually hurt performance.<br>
<br>
2. Your test is flawed because dd uses caching unless you specify dsync.<br>
Set that and also use a larger write file (I'd recommend minimum of 2x RAM).<br>
<br>
3. You need to also test the network connection between the two nodes.<br>
Use sperf or similar to ensure you are actually getting the network link<br>
speed you expect.<br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
--<br>
Digimer<br>
Papers and Projects: <a href="https://alteeve.com/w/" rel="noreferrer" target="_blank">https://alteeve.com/w/</a><br>
"I am, somehow, less interested in the weight and convolutions of<br>
Einstein’s brain than in the near certainty that people of equal talent<br>
have lived and died in cotton fields and sweatshops." - Stephen Jay Gould<br>
</font></span></blockquote></div><br></div>