<p dir="ltr">Have you tried on-congestion pull-ahead in protocol A?</p>
<p dir="ltr">J.R. Lillard<br>
System / Network Admin<br>
Web Programmer<br>
Golden Heritage Foods</p>
<div class="gmail_quote">On Nov 10, 2013 1:11 PM, "Bram Matthys" <<a href="mailto:syzop@vulnscan.org">syzop@vulnscan.org</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi all,<br>
<br>
Any ideas? I'm essentially blocked now.. I can't go into production with<br>
these write speeds.<br>
<br>
Am I the only one who experiences a >60% degradation of write speed when the<br>
two nodes are connected to each other? With full disk speed only in<br>
disconnected state?<br>
<br>
Regards,<br>
<br>
Bram.<br>
<br>
Bram Matthys wrote, on 3-11-2013 11:26:<br>
> Hi,<br>
><br>
> Using a RAID10 array now and (following Arnolds advice) the metadata is now<br>
> on another (SSD) disk.<br>
> Write speeds are still bad, but are ONLY low when both nodes are connected.<br>
><br>
> root@st2:/data# sync; echo 3 >/proc/sys/vm/drop_caches ;sync; dd<br>
> if=/dev/zero of=5G bs=1M count=5000; rm -f 5G<br>
> 5000+0 records in<br>
> 5000+0 records out<br>
> 5242880000 bytes (5.2 GB) copied, 57.1802 s, 91.7 MB/s<br>
><br>
> root@st2:/data# drbdadm disconnect data<br>
> root@st2:/data# sync; echo 3 >/proc/sys/vm/drop_caches ;sync; dd<br>
> if=/dev/zero of=5G bs=1M count=5000; rm -f 5G<br>
> 5000+0 records in<br>
> 5000+0 records out<br>
> 5242880000 bytes (5.2 GB) copied, 21.4724 s, 244 MB/s<br>
><br>
> When I (re)connect data again, the resync speed is fine:<br>
> st1# cat /proc/drbd<br>
> version: 8.3.11 (api:88/proto:86-96)<br>
> srcversion: F937DCB2E5D83C6CCE4A6C9<br>
><br>
> 1: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----<br>
> ns:0 nr:5121168 dw:5121168 dr:2010340 al:0 bm:133 lo:0 pe:206 ua:0 ap:0<br>
> ep:1 wo:d oos:3130232<br>
> [======>.............] sync'ed: 38.9% (3056/4996)Mfinish: 0:00:14<br>
> speed: 220,760 (220,760) want: 256,000 K/sec<br>
><br>
> And, as mentioned earlier, netperf achieves near-10gE speed:<br>
> # netperf -H 192.168.220.1<br>
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.220.1<br>
> (192.168.220.1) port 0 AF_INET : demo<br>
> Recv Send Send<br>
> Socket Socket Message Elapsed<br>
> Size Size Size Time Throughput<br>
> bytes bytes bytes secs. 10^6bits/sec<br>
><br>
> 87380 16384 16384 10.00 9897.79<br>
> (same in both directions)<br>
><br>
> It doesn't matter whether I make server st1 or st2 primary (so either<br>
> server) and do the tests... with both they achieve around 240 MB/s write<br>
> speed in disconnected mode and around 80-90 MB/s when connected, so ~ 1/3rd.<br>
><br>
> Any ideas?<br>
><br>
> # drbdadm dump<br>
> # /etc/drbd.conf<br>
> common {<br>
> protocol C;<br>
> startup {<br>
> degr-wfc-timeout 120;<br>
> wfc-timeout 120;<br>
> }<br>
> }<br>
><br>
> # resource data on st2: not ignored, not stacked<br>
> resource data {<br>
> on st1 {<br>
> device /dev/drbd1 minor 1;<br>
> disk /dev/md4;<br>
> address ipv4 <a href="http://192.168.220.1:7789" target="_blank">192.168.220.1:7789</a>;<br>
> meta-disk /dev/md3 [0];<br>
> }<br>
> on st2 {<br>
> device /dev/drbd1 minor 1;<br>
> disk /dev/md4;<br>
> address ipv4 <a href="http://192.168.220.2:7789" target="_blank">192.168.220.2:7789</a>;<br>
> meta-disk /dev/md3 [0];<br>
> }<br>
> net {<br>
> data-integrity-alg sha256;<br>
> max-buffers 8000;<br>
> max-epoch-size 8000;<br>
> sndbuf-size 512k;<br>
> }<br>
> disk {<br>
> no-disk-barrier;<br>
> no-disk-flushes;<br>
> }<br>
> syncer {<br>
> csums-alg sha256;<br>
> rate 250M;<br>
> }<br>
> }<br>
><br>
> NOTE: I was using protocol A without no-disk-* before, just using<br>
> C/no-disk-* to see if it made any significant difference: answer is no.<br>
><br>
> root@st2:/data# uname -a<br>
> Linux st2 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1+deb7u1 x86_64 GNU/Linux<br>
> root@st2:/data# cat /proc/drbd<br>
> version: 8.3.11 (api:88/proto:86-96)<br>
> srcversion: F937DCB2E5D83C6CCE4A6C9<br>
><br>
> 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----<br>
> ns:5121332 nr:0 dw:15358180 dr:5130289 al:3773 bm:377 lo:0 pe:0 ua:0<br>
> ap:0 ep:1 wo:d oos:0<br>
><br>
> root@st2:/data# dpkg --list|grep drbd<br>
> ii drbd8-utils 2:8.3.13-2 amd64<br>
> RAID 1 over tcp/ip for Linux utilities<br>
><br>
> If you need anything else, just let me know.<br>
><br>
> Thanks again,<br>
><br>
> Bram.<br>
><br>
> Arnold Krille wrote, on 7-10-2013 1:37:<br>
>> Hi,<br>
>><br>
>> On Sun, 06 Oct 2013 18:54:12 +0200 Bram Matthys <<a href="mailto:syzop@vulnscan.org">syzop@vulnscan.org</a>><br>
>> wrote:<br>
>>> I'm currently testing DRBD and am having write performance problems.<br>
>>> On the local raid array I achieve 124MB/s, but with DRBD I get only<br>
>>> 41MB/s out of it, or if the secondary node is down (to rule out<br>
>>> network issues) then 53MB/s at best.<br>
>>> Tried protocol A / B / C, with and without no-disk-barriers and<br>
>>> no-disk-flushes, but this didn't change much (only +/- 2MB/s<br>
>>> difference).<br>
>> <snip><br>
>><br>
>> Yep, internal meta-disk: write block, seek to end, write log, seek to<br>
>> front, write block, seek to end, write log...<br>
>><br>
>> Put the meta-disk on a different hd and watch your write rate go up.<br>
>><br>
>> - Arnold<br>
>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> drbd-user mailing list<br>
>> <a href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br>
>> <a href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
>><br>
><br>
><br>
<br>
<br>
--<br>
Bram Matthys<br>
Software developer/IT consultant <a href="mailto:syzop@vulnscan.org">syzop@vulnscan.org</a><br>
Website: <a href="http://www.vulnscan.org" target="_blank">www.vulnscan.org</a><br>
PGP key: <a href="http://www.vulnscan.org/pubkey.asc" target="_blank">www.vulnscan.org/pubkey.asc</a><br>
PGP fp: EBCA 8977 FCA6 0AB0 6EDB 04A7 6E67 6D45 7FE1 99A6<br>
<br>
<br>
_______________________________________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
</blockquote></div>