Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
By performance improvements, I was talking about this: https://blogs.linbit.com/p/469/843-random-writes-faster/ 8.4 is significantly faster than 8.3 As of today - I'm getting pretty darn good write speeds through my whole KVM->DRBD->IPoIB->LV->RAID stack. So I will continue to use the regular 'ipv4' connection setting. [root at cinhpcdev2 ~]# dd if=/dev/zero of=/dev/vg_cinhpcdev2/test bs=1M count=4096 oflag=direct 4096+0 records in 4096+0 records out 4294967296 bytes (4.3 GB) copied, 11.6276 s, 369 MB/s That is still less than 1/2 of what the controllers and interconnect are capable of, but with tuning some of the settings I may be able to get that higher, there is likely some overhead in the VirtIO disk controller the VM is using. If SDP is no longer supported in the 8.4.x series - the documentation should likely be updated to reflect that fact, and the transport option disabled. https://drbd.linbit.com/users-guide/s-replication-transports.html Also - If you are curious: A few lines from the NPtcp test of regular TCP versus SDP over the Infiniband layer. Note, these are only a few lines from the 124 different tests. [root at cinhpcvm05 ~]# LD_PRELOAD=libsdp.so.1 NPtcp -h cinhpcvm06i 8: 16 bytes 12047 times --> 32.20 Mbps in 3.79 usec 20: 64 bytes 12772 times --> 128.81 Mbps in 3.79 usec 44: 1024 bytes 10746 times --> 1688.06 Mbps in 4.63 usec 74: 32768 bytes 2881 times --> 14410.92 Mbps in 17.35 usec 80: 65536 bytes 2000 times --> 20000.21 Mbps in 25.00 usec 105: 1048579 bytes 137 times --> 21812.54 Mbps in 366.76 usec Versus no SDP tcp [root at cinhpcvm05 ~]# NPtcp -h cinhpcvm06i 8: 16 bytes 1405 times --> 4.85 Mbps in 25.15 usec 20: 64 bytes 1522 times --> 14.39 Mbps in 33.94 usec 44: 1024 bytes 1467 times --> 220.86 Mbps in 35.37 usec 74: 32768 bytes 626 times --> 3721.55 Mbps in 67.18 usec 80: 65536 bytes 587 times --> 5938.40 Mbps in 84.20 usec 105: 1048579 bytes 56 times --> 9711.10 Mbps in 823.80 usec The plan is to use DRBD in our production environment to add some more reliability to our virtual machines. If IPv4 is just flat out more supported and more stable then I will just use that and tune it the best I can. On Thu, Nov 19, 2015 at 9:16 AM, Lars Ellenberg <lars.ellenberg at linbit.com> wrote: > On Tue, Nov 17, 2015 at 11:50:40PM -0500, Chuck Bredestege wrote: > > I found this thread which is very close to the issue I am seeing: > > http://lists.linbit.com/pipermail/drbd-user/2015-October/022484.html > > Is that so. > And, did the patch I posted there help, or change behaviour in any way? > > > If I change the resource file to use 'ipv4' instead of 'sdp' it works > just > > fine. I get about 350MB/s throughput which is very low considering the > > raid controller consistently gets about 900MB/s - but that's for a > > different conversation. I would like to get the SDP protocol working > since > > the NPtcp benchmark from netpipe yields 2x the bandwidth and 1/2 the > > latency as using TCP/IPoIB alone. (20GB/s and 3.1usec). > > > > > > Again - SDP works in 8.3 without complaint - it's just slow. > > There is no reason to believe SDP would be "faster" with 8.4, > even if it worked. It would still be the very same SDP. > > In my experience, DRBD + SDP does not help with throughput or latency, > not really anyways. DRBD + tuned-for-DRBD SDP would only help to > reduce CPU usage compared to IPoIB mode. > > > I saw the patch file at the end of the thread I linked above, > > I applied it and there was no change. > > Too bad. > > > Any help would be greatly appreciated. > > Apparently SDP does not work in DRBD 8.4. > I'm not aware of any customer trying to use it, either. > > The underlying problem is that at some point we changed from > blocking accept() to using ->sk_state_change() callbacks. > > Which SDP does not implement. > So we never notice that a connection is actually established, > it will always appear to "time out". > > You could still change the first kernel accept into a blocking one, > in the patch referenced above. > > - err = kernel_accept(ad->s_listen, &s_estab, O_NONBLOCK); > + err = kernel_accept(ad->s_listen, &s_estab, 0); > > And hope that this does not block forever, > and maybe even works similar to how it used to work with drbd 8.3. > > At least that should give you a starting point... > > -- > : Lars Ellenberg > : http://www.LINBIT.com | Your Way to High Availability > : DRBD, Linux-HA and Pacemaker support and consulting > > DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. > __ > please don't Cc me, but send to list -- I'm subscribed > _______________________________________________ > drbd-user mailing list > drbd-user at lists.linbit.com > http://lists.linbit.com/mailman/listinfo/drbd-user > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20151119/dc84ffc0/attachment.htm>