[DRBD-user] SDP Connection problems with DRBD 8.4.6

Chuck Bredestege cbredestege at gmail.com
Thu Nov 19 16:06:45 CET 2015

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.


By performance improvements, I was talking about this:
https://blogs.linbit.com/p/469/843-random-writes-faster/
8.4 is significantly faster than 8.3

As of today - I'm getting pretty darn good write speeds through my whole
KVM->DRBD->IPoIB->LV->RAID stack.  So I will continue to use the regular
'ipv4' connection setting.

[root at cinhpcdev2 ~]# dd if=/dev/zero of=/dev/vg_cinhpcdev2/test bs=1M
count=4096 oflag=direct
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB) copied, 11.6276 s, 369 MB/s

That is still less than 1/2 of what the controllers and interconnect are
capable of, but with tuning some of the settings I may be able to get that
higher, there is likely some overhead in the VirtIO disk controller the VM
is using.

If SDP is no longer supported in the 8.4.x series - the documentation
should likely be updated to reflect that fact, and the transport option
disabled.

https://drbd.linbit.com/users-guide/s-replication-transports.html

Also - If you are curious:

A few lines from the NPtcp test of regular TCP versus SDP over the
Infiniband layer.  Note, these are only a few lines from the 124 different
tests.

[root at cinhpcvm05 ~]# LD_PRELOAD=libsdp.so.1 NPtcp -h cinhpcvm06i
8:      16 bytes  12047 times -->     32.20 Mbps in       3.79 usec
20:      64 bytes  12772 times -->    128.81 Mbps in       3.79 usec
44:    1024 bytes  10746 times -->   1688.06 Mbps in       4.63 usec
74:   32768 bytes   2881 times -->  14410.92 Mbps in      17.35 usec
80:   65536 bytes   2000 times -->  20000.21 Mbps in      25.00 usec
105: 1048579 bytes    137 times -->  21812.54 Mbps in     366.76 usec

Versus no  SDP tcp

[root at cinhpcvm05 ~]# NPtcp -h cinhpcvm06i
  8:      16 bytes   1405 times -->      4.85 Mbps in      25.15 usec
 20:      64 bytes   1522 times -->     14.39 Mbps in      33.94 usec
 44:    1024 bytes   1467 times -->    220.86 Mbps in      35.37 usec
 74:   32768 bytes    626 times -->   3721.55 Mbps in      67.18 usec
 80:   65536 bytes    587 times -->   5938.40 Mbps in      84.20 usec
105: 1048579 bytes     56 times -->   9711.10 Mbps in     823.80 usec


The plan is to use DRBD in our production environment to add some more
reliability to our virtual machines.  If IPv4 is just flat out more
supported and more stable then I will just use that and tune it the best I
can.



On Thu, Nov 19, 2015 at 9:16 AM, Lars Ellenberg <lars.ellenberg at linbit.com>
wrote:

> On Tue, Nov 17, 2015 at 11:50:40PM -0500, Chuck Bredestege wrote:
> > I found this thread which is very close to the issue I am seeing:
> > http://lists.linbit.com/pipermail/drbd-user/2015-October/022484.html
>
> Is that so.
> And, did the patch I posted there help, or change behaviour in any way?
>
> > If I change the resource file to use 'ipv4' instead of 'sdp' it works
> just
> > fine.  I get about 350MB/s throughput which is very low considering the
> > raid controller consistently gets about 900MB/s - but that's for a
> > different conversation.  I would like to get the SDP protocol working
> since
> > the NPtcp benchmark from netpipe yields 2x the bandwidth and 1/2 the
> > latency as using TCP/IPoIB alone.  (20GB/s and 3.1usec).
> >
> >
> > Again - SDP works in 8.3 without complaint - it's just slow.
>
> There is no reason to believe SDP would be "faster" with 8.4,
> even if it worked. It would still be the very same SDP.
>
> In my experience, DRBD + SDP does not help with throughput or latency,
> not really anyways. DRBD + tuned-for-DRBD SDP would only help to
> reduce CPU usage compared to IPoIB mode.
>
> > I saw the patch file at the end of the thread I linked above,
> > I applied it and there was no change.
>
> Too bad.
>
> > Any help would be greatly appreciated.
>
> Apparently SDP does not work in DRBD 8.4.
> I'm not aware of any customer trying to use it, either.
>
> The underlying problem is that at some point we changed from
> blocking accept() to using ->sk_state_change() callbacks.
>
> Which SDP does not implement.
> So we never notice that a connection is actually established,
> it will always appear to "time out".
>
> You could still change the first kernel accept into a blocking one,
> in the patch referenced above.
>
> - err = kernel_accept(ad->s_listen, &s_estab, O_NONBLOCK);
> + err = kernel_accept(ad->s_listen, &s_estab, 0);
>
> And hope that this does not block forever,
> and maybe even works similar to how it used to work with drbd 8.3.
>
> At least that should give you a starting point...
>
> --
> : Lars Ellenberg
> : http://www.LINBIT.com | Your Way to High Availability
> : DRBD, Linux-HA  and  Pacemaker support and consulting
>
> DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
> __
> please don't Cc me, but send to list   --   I'm subscribed
> _______________________________________________
> drbd-user mailing list
> drbd-user at lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20151119/dc84ffc0/attachment.htm>


More information about the drbd-user mailing list