Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
I found this thread which is very close to the issue I am seeing:
http://lists.linbit.com/pipermail/drbd-user/2015-October/022484.html
I am unable to get the SDP protocol working with DRBD 8.4.6 and drbd utils
8.9.4.
Both systems are identical
RHEL6.7
kernal 2.6.32_573.3.1.el6.x86_64
HP DL380 Gen8's with Mellanox Connect-X 3 class 56gbps FDR Infiniband cards
OFED 3.1
libsdp.so.1 is available
ib_sdp.ko is loaded.
I compiled the drbd kernal module and utilities from source.
the DRBD 8.3 module works properly, but the performance was lackluster and
I'm hoping to get better throughput in 8.4.
using the SDP aware "netpipe" benchmark NPtcp I can achieve upwards to
20GB/s with larger packet sizes.
When I try to turn up a DRBD resource using the SDP protocol both systems
sit in a WFConnection state and it appears that the systems are repeatedly
connecting to each other but not working.
[root at cinhpcvm06 drbd.d]# cat /proc/drbd
version: 8.4.6 (api:1/proto:86-101)
GIT-hash: 833d830e0152d1e457fa7856e71e11248ccf3f70 build by
****@cinhpcdev3, 2015-11-17 22:10:29
1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:0
using the utility "sdpnetstat" which is part of the ofed series I can see
the active connections
[root at cinhpcvm06 drbd.d]# sdpnetstat -S
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
sdp 0 0 cinhpcvm06i:48081 cinhpcvm05i:7791 FIN_WAIT1
sdp 0 0 cinhpcvm06i:53608 cinhpcvm05i:7791 FIN_WAIT1
sdp 0 0 cinhpcvm06i:39639 cinhpcvm05i:7791 ESTABLISHED
sdp 0 0 cinhpcvm06i:7791 cinhpcvm05i:38979 ESTABLISHED
If I change the resource file to use 'ipv4' instead of 'sdp' it works just
fine. I get about 350MB/s throughput which is very low considering the
raid controller consistently gets about 900MB/s - but that's for a
different conversation. I would like to get the SDP protocol working since
the NPtcp benchmark from netpipe yields 2x the bandwidth and 1/2 the
latency as using TCP/IPoIB alone. (20GB/s and 3.1usec).
Again - SDP works in 8.3 without complaint - it's just slow.
I saw the patch file at the end of the thread I linked above, I applied it
and there was no change.
Any help would be greatly appreciated.
Thanks in advance,
~Chuck Bredestege
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20151117/0f6e33f4/attachment.htm>