<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body text="#000000" bgcolor="#FFFFFF"><div dir="auto">Try to remove the following:<br><br></div>
<div dir="auto">c-fill-target 24M;<br></div>
<div dir="auto">c-min-rate 80M;<br></div>
<div dir="auto">c-max-rate 720M;<br><br></div>
<div dir="auto">sndbuf-size 1024k;<br></div>
<div dir="auto">rcvbuf-size 2048k;<br><br></div>
<div dir="auto">Then gradually increase max-buffers from 4K to 12K checking its impact to the sync speed. Make sure you have the same config on both nodes and apply the changes with "drbdadm adjust all" on both nodes too. <br><br></div>
<div dir="auto"><!-- tmjah_g_1299s -->//OE<!-- tmjah_g_1299e --></div>
<div class="gmail_quote" >On Oct 11, 2018, at 21:43, Adam Weremczuk <<a href="mailto:adamw@matrixscience.com" target="_blank">adamw@matrixscience.com</a>> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<p>Hi Oleksiy,</p>
<p>Here is my entire current config:<br>
</p>
<p>global { usage-count no; }<br>
common { protocol C; }<br>
resource r0 {<br>
disk {<br>
on-io-error detach;<br>
no-disk-flushes ;<br>
no-disk-barrier;<br>
c-plan-ahead 0;<br>
c-fill-target 24M;<br>
c-min-rate 80M;<br>
c-max-rate 720M;<br>
}<br>
net {<br>
max-buffers 36k;<br>
sndbuf-size 1024k;<br>
rcvbuf-size 2048k;<br>
}<br>
on node1 {<br>
device /dev/drbd0;<br>
disk /dev/sdb1;<br>
address 192.168.200.1:7788;<br>
meta-disk internal;<br>
}<br>
on node2 {<br>
device /dev/drbd0;<br>
disk /dev/sdb1;<br>
address 192.168.200.2:7788;<br>
meta-disk internal;<br>
}<br>
}<br>
</p>
<p>...and the speed is still dreadfully slow, even though the link
can easily do 150MB/s:<br>
<br>
cat /proc/drbd<br>
version: 8.4.10 (api:1/proto:86-101)<br>
srcversion: 17A0C3A0AF9492ED4B9A418 <br>
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C
r-----<br>
ns:46712 nr:0 dw:0 dr:47528 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1
wo:d oos:3898265768<br>
[>....................] sync'ed: 0.1% (3806900/3806944)M<br>
finish: 3867:19:37 speed: 264 (260) K/sec</p>
<p>Anything else to try before giving up on DRBD?<br>
<br>
Thanks,<br>
Adam<br>
</p>
<br>
<div class="moz-cite-prefix">On 11/10/18 07:45, Oleksiy Evin wrote:<br>
</div>
<blockquote type="cite" cite="mid:c380fd0c7c45c69f7b6f5110bedca830cdfa2db6.camel@onefc.com">
<div>You may try to disable dynamic sync rate by
setting "c-plan-ahead" to 0 and increase "max-buffers". That's
the only way which helped me to get reasonable sync rate with
100GigE connection.</div>
<div><br>
</div>
<div>net {</div>
<div> ...</div>
<div> max-buffers 32K;</div>
<div> # max-epoch-size 18K;</div>
<div>}</div>
<div><br>
</div>
<div>disk {</div>
<div> ... </div>
<div> c-plan-ahead 0;</div>
<div>}</div>
<div><br>
</div>
<div>You can find some more info here: </div>
<div><a href="https://serverfault.com/questions/740311/drbd-terrible-sync-performance-on-10gige/740370" moz-do-not-send="true">https://serverfault.com/questions/740311/drbd-terrible-sync-performance-on-10gige/740370</a></div>
<div><br>
</div>
<div><br>
</div>
<div>//OE</div>
<div><br>
</div>
<div>-----Original Message-----</div>
<div><b>From</b>: Adam Weremczuk <<a href="mailto:Adam%20Weremczuk%20%3cadamw@matrixscience.com%3e" moz-do-not-send="true">adamw@matrixscience.com</a>></div>
<div><b>To</b>: <a href="mailto:drbd-user@lists.linbit.com" moz-do-not-send="true">drbd-user@lists.linbit.com</a></div>
<div><b>Subject</b>: [DRBD-user] slow sync speed</div>
<div><b>Date</b>: Wed, 10 Oct 2018 14:57:02 +0100</div>
<div><br>
</div>
<pre>Hi all,</pre>
<pre></pre>
<pre>I'm trying out DRBD Pacemaker HA Cluster on Proxmox 5.2</pre>
<pre></pre>
<pre>I have 2 identical servers connected with 2 x 1 Gbps links in bond_mode </pre>
<pre>balance-rr.</pre>
<pre></pre>
<pre>The bond is working fine; I get a transfer rate of 150 MB/s with scp.</pre>
<pre></pre>
<pre>Following this guide: </pre>
<pre><a href="https://www.theurbanpenguin.com/drbd-pacemaker-ha-cluster-ubuntu-16-04/" moz-do-not-send="true">https://www.theurbanpenguin.com/drbd-pacemaker-ha-cluster-ubuntu-16-04/</a> </pre>
<pre>was going smoothly up until:</pre>
<pre></pre>
<pre>drbdadm -- --overwrite-data-of-peer primary r0/0</pre>
<pre></pre>
<pre>cat /proc/drbd</pre>
<pre>version: 8.4.10 (api:1/proto:86-101)</pre>
<pre>srcversion: 17A0C3A0AF9492ED4B9A418</pre>
<pre> 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----</pre>
<pre> ns:10944 nr:0 dw:0 dr:10992 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f </pre>
<pre>oos:3898301536</pre>
<pre> [>....................] sync'ed: 0.1% (3806932/3806944)M</pre>
<pre> finish: 483:25:13 speed: 2,188 (2,188) K/sec</pre>
<pre></pre>
<pre>The transfer rate is horribly slow and at this pace it's going to take </pre>
<pre>20 days for two 4 TB volumes to sync!</pre>
<pre></pre>
<pre>That's almost 15 times slower comparing with the guide video (8:30): </pre>
<pre><a href="https://www.youtube.com/watch?v=WQGi8Nf0kVc" moz-do-not-send="true">https://www.youtube.com/watch?v=WQGi8Nf0kVc</a></pre>
<pre></pre>
<pre>The volumes have been zeroed and contain no live data yet.</pre>
<pre></pre>
<pre>My sdb disks are logical drives (hardware RAID) set up as RAID50 with </pre>
<pre>the defaults:</pre>
<pre></pre>
<pre>Strip size: 128 KB</pre>
<pre>Access policy: RW</pre>
<pre>Read policy: Normal</pre>
<pre>Write policy: Write Back with BBU</pre>
<pre>IO policy: Direct</pre>
<pre>Drive Cache: Disable</pre>
<pre>Disable BGI: No</pre>
<pre></pre>
<pre>Performance looks good when tested with hdparm:</pre>
<pre></pre>
<pre>hdparm -tT /dev/sdb1</pre>
<pre></pre>
<pre>/dev/sdb1:</pre>
<pre> Timing cached reads: 15056 MB in 1.99 seconds = 7550.46 MB/sec</pre>
<pre> Timing buffered disk reads: 2100 MB in 3.00 seconds = 699.81 MB/sec</pre>
<pre></pre>
<pre>The volumes have been zeroed and contain no live data yet.</pre>
<pre></pre>
<pre>It seems to be a problem with default DRBD settings.</pre>
<pre></pre>
<pre>Can anybody recommend optimal tweaks specific to my environment?</pre>
<pre></pre>
<pre>Regards,</pre>
<pre>Adam</pre>
<pre></pre>
<pre>_______________________________________________</pre>
<pre>drbd-user mailing list</pre>
<pre><a href="mailto:drbd-user@lists.linbit.com" moz-do-not-send="true">drbd-user@lists.linbit.com</a></pre>
<pre><a href="http://lists.linbit.com/mailman/listinfo/drbd-user" moz-do-not-send="true">http://lists.linbit.com/mailman/listinfo/drbd-user</a></pre>
<pre></pre>
</blockquote>
<br>
</blockquote></div></body></html>