<div dir="ltr">By performance improvements, I was talking about this: <a href="https://blogs.linbit.com/p/469/843-random-writes-faster/">https://blogs.linbit.com/p/469/843-random-writes-faster/</a> <div>8.4 is significantly faster than 8.3</div><div><br></div><div>As of today - I'm getting pretty darn good write speeds through my whole KVM->DRBD->IPoIB->LV->RAID stack. So I will continue to use the regular 'ipv4' connection setting.</div><div><br></div><div><div>[root@cinhpcdev2 ~]# dd if=/dev/zero of=/dev/vg_cinhpcdev2/test bs=1M count=4096 oflag=direct</div><div>4096+0 records in</div><div>4096+0 records out</div><div>4294967296 bytes (4.3 GB) copied, 11.6276 s, 369 MB/s</div></div><div><br></div><div>That is still less than 1/2 of what the controllers and interconnect are capable of, but with tuning some of the settings I may be able to get that higher, there is likely some overhead in the VirtIO disk controller the VM is using.</div><div><br></div><div>If SDP is no longer supported in the 8.4.x series - the documentation should likely be updated to reflect that fact, and the transport option disabled.</div><div><br></div><div><a href="https://drbd.linbit.com/users-guide/s-replication-transports.html">https://drbd.linbit.com/users-guide/s-replication-transports.html</a><br></div><div><br></div><div>Also - If you are curious:</div><div><br></div><div>A few lines from the NPtcp test of regular TCP versus SDP over the Infiniband layer. Note, these are only a few lines from the 124 different tests.</div><div><br></div><div>[root@cinhpcvm05 ~]# LD_PRELOAD=libsdp.so.1 NPtcp -h cinhpcvm06i<br></div><div>8: 16 bytes 12047 times --> 32.20 Mbps in 3.79 usec<br></div><div>20: 64 bytes 12772 times --> 128.81 Mbps in 3.79 usec<br></div><div>44: 1024 bytes 10746 times --> 1688.06 Mbps in 4.63 usec<br></div><div>74: 32768 bytes 2881 times --> 14410.92 Mbps in 17.35 usec<br></div><div>80: 65536 bytes 2000 times --> 20000.21 Mbps in 25.00 usec<br></div><div>105: 1048579 bytes 137 times --> 21812.54 Mbps in 366.76 usec<br></div><div><br></div><div>Versus no SDP tcp</div><div><br></div><div>[root@cinhpcvm05 ~]# NPtcp -h cinhpcvm06i<br></div><div> 8: 16 bytes 1405 times --> 4.85 Mbps in 25.15 usec<br></div><div> 20: 64 bytes 1522 times --> 14.39 Mbps in 33.94 usec<br></div><div> 44: 1024 bytes 1467 times --> 220.86 Mbps in 35.37 usec<br></div><div> 74: 32768 bytes 626 times --> 3721.55 Mbps in 67.18 usec<br></div><div> 80: 65536 bytes 587 times --> 5938.40 Mbps in 84.20 usec<br></div><div>105: 1048579 bytes 56 times --> 9711.10 Mbps in 823.80 usec<br></div><div><br></div><div><br></div><div>The plan is to use DRBD in our production environment to add some more reliability to our virtual machines. If IPv4 is just flat out more supported and more stable then I will just use that and tune it the best I can.</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 19, 2015 at 9:16 AM, Lars Ellenberg <span dir="ltr"><<a href="mailto:lars.ellenberg@linbit.com" target="_blank">lars.ellenberg@linbit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Tue, Nov 17, 2015 at 11:50:40PM -0500, Chuck Bredestege wrote:<br>
> I found this thread which is very close to the issue I am seeing:<br>
> <a href="http://lists.linbit.com/pipermail/drbd-user/2015-October/022484.html" rel="noreferrer" target="_blank">http://lists.linbit.com/pipermail/drbd-user/2015-October/022484.html</a><br>
<br>
</span>Is that so.<br>
And, did the patch I posted there help, or change behaviour in any way?<br>
<span class=""><br>
> If I change the resource file to use 'ipv4' instead of 'sdp' it works just<br>
> fine. I get about 350MB/s throughput which is very low considering the<br>
> raid controller consistently gets about 900MB/s - but that's for a<br>
> different conversation. I would like to get the SDP protocol working since<br>
> the NPtcp benchmark from netpipe yields 2x the bandwidth and 1/2 the<br>
> latency as using TCP/IPoIB alone. (20GB/s and 3.1usec).<br>
><br>
><br>
> Again - SDP works in 8.3 without complaint - it's just slow.<br>
<br>
</span>There is no reason to believe SDP would be "faster" with 8.4,<br>
even if it worked. It would still be the very same SDP.<br>
<br>
In my experience, DRBD + SDP does not help with throughput or latency,<br>
not really anyways. DRBD + tuned-for-DRBD SDP would only help to<br>
reduce CPU usage compared to IPoIB mode.<br>
<span class=""><br>
> I saw the patch file at the end of the thread I linked above,<br>
> I applied it and there was no change.<br>
<br>
</span>Too bad.<br>
<span class=""><br>
> Any help would be greatly appreciated.<br>
<br>
</span>Apparently SDP does not work in DRBD 8.4.<br>
I'm not aware of any customer trying to use it, either.<br>
<br>
The underlying problem is that at some point we changed from<br>
blocking accept() to using ->sk_state_change() callbacks.<br>
<br>
Which SDP does not implement.<br>
So we never notice that a connection is actually established,<br>
it will always appear to "time out".<br>
<br>
You could still change the first kernel accept into a blocking one,<br>
in the patch referenced above.<br>
<br>
- err = kernel_accept(ad->s_listen, &s_estab, O_NONBLOCK);<br>
+ err = kernel_accept(ad->s_listen, &s_estab, 0);<br>
<br>
And hope that this does not block forever,<br>
and maybe even works similar to how it used to work with drbd 8.3.<br>
<br>
At least that should give you a starting point...<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
: Lars Ellenberg<br>
: <a href="http://www.LINBIT.com" rel="noreferrer" target="_blank">http://www.LINBIT.com</a> | Your Way to High Availability<br>
: DRBD, Linux-HA and Pacemaker support and consulting<br>
<br>
DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.<br>
__<br>
please don't Cc me, but send to list -- I'm subscribed<br>
_______________________________________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" rel="noreferrer" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
</font></span></blockquote></div><br></div>