<div dir="ltr">I found this thread which is very close to the issue I am seeing:<div><a href="http://lists.linbit.com/pipermail/drbd-user/2015-October/022484.html">http://lists.linbit.com/pipermail/drbd-user/2015-October/022484.html</a><br></div><div><br></div><div>I am unable to get the SDP protocol working with DRBD 8.4.6 and drbd utils 8.9.4.</div><div><br></div><div>Both systems are identical</div><div>RHEL6.7</div><div>kernal 2.6.32_573.3.1.el6.x86_64</div><div>HP DL380 Gen8's with Mellanox Connect-X 3 class 56gbps FDR Infiniband cards<br></div><div>OFED 3.1<br></div><div>libsdp.so.1 is available</div><div>ib_sdp.ko is loaded.<br></div><div><br></div><div>I compiled the drbd kernal module and utilities from source.</div><div><br></div><div>the DRBD 8.3 module works properly, but the performance was lackluster and I'm hoping to get better throughput in 8.4.</div><div><br></div><div>using the SDP aware "netpipe" benchmark NPtcp I can achieve upwards to 20GB/s with larger packet sizes.</div><div><br></div><div>When I try to turn up a DRBD resource using the SDP protocol both systems sit in a WFConnection state and it appears that the systems are repeatedly connecting to each other but not working.</div><div><br></div><div><p style="margin:0in;font-family:'Courier New';font-size:11pt">[root@cinhpcvm06 drbd.d]# cat /proc/drbd</p>
<p style="margin:0in;font-family:'Courier New';font-size:11pt">version: 8.4.6
(api:1/proto:86-101)</p>
<p style="margin:0in;font-family:'Courier New';font-size:11pt">GIT-hash:
833d830e0152d1e457fa7856e71e11248ccf3f70 build by ****@cinhpcdev3,
2015-11-17 22:10:29</p>
<p style="margin:0in;font-family:'Courier New';font-size:11pt"> </p>
<p style="margin:0in;font-family:'Courier New';font-size:11pt"> 1: cs:WFConnection ro:Secondary/Unknown
ds:UpToDate/DUnknown C r-----</p>
<p style="margin:0in;font-family:'Courier New';font-size:11pt"> ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0
ua:0 ap:0 ep:1 wo:d oos:0</p><p style="margin:0in;font-family:'Courier New';font-size:11pt"><br></p><p style="margin:0in;font-family:'Courier New';font-size:11pt">using the utility "sdpnetstat" which is part of the ofed series I can see the active connections</p><p style="margin:0in;font-family:'Courier New';font-size:11pt"><br></p><p style="margin:0in;font-family:'Courier New';font-size:11pt">[root@cinhpcvm06
drbd.d]# sdpnetstat -S</p><p style="margin:0in;font-family:'Courier New';font-size:11pt">Active
Internet connections (w/o servers)</p><p style="margin:0in;font-family:'Courier New';font-size:11pt">Proto Recv-Q
Send-Q Local Address Foreign
Address State</p><p style="margin:0in;font-family:'Courier New';font-size:11pt">sdp 0
0 cinhpcvm06i:48081 cinhpcvm05i:7791 FIN_WAIT1</p><p style="margin:0in;font-family:'Courier New';font-size:11pt">sdp 0
0 cinhpcvm06i:53608 cinhpcvm05i:7791 FIN_WAIT1</p><p style="margin:0in;font-family:'Courier New';font-size:11pt">sdp 0
0 cinhpcvm06i:39639 cinhpcvm05i:7791 ESTABLISHED</p><p style="margin:0in;font-family:'Courier New';font-size:11pt">sdp 0
0 cinhpcvm06i:7791 cinhpcvm05i:38979 ESTABLISHED</p><p style="margin:0in;font-family:'Courier New';font-size:11pt">
</p><p style="margin:0in;font-family:'Courier New';font-size:11pt"> </p><p style="margin:0in;font-family:'Courier New';font-size:11pt"><br></p><p style="margin:0in;font-family:'Courier New';font-size:11pt">If I change the resource file to use 'ipv4' instead of 'sdp' it works just fine. I get about 350MB/s throughput which is very low considering the raid controller consistently gets about 900MB/s - but that's for a different conversation. I would like to get the SDP protocol working since the NPtcp benchmark from netpipe yields 2x the bandwidth and 1/2 the latency as using TCP/IPoIB alone. (20GB/s and 3.1usec).</p><p style="margin:0in;font-family:'Courier New';font-size:11pt"><br></p><p style="margin:0in;font-family:'Courier New';font-size:11pt">Again - SDP works in 8.3 without complaint - it's just slow.</p><p style="margin:0in;font-family:'Courier New';font-size:11pt"><br></p><p style="margin:0in;font-family:'Courier New';font-size:11pt">I saw the patch file at the end of the thread I linked above, I applied it and there was no change.</p><p style="margin:0in;font-family:'Courier New';font-size:11pt"><br></p><p style="margin:0in;font-family:'Courier New';font-size:11pt">Any help would be greatly appreciated.</p><p style="margin:0in;font-family:'Courier New';font-size:11pt"><br></p><p style="margin:0in;font-family:'Courier New';font-size:11pt">Thanks in advance,</p><p style="margin:0in;font-family:'Courier New';font-size:11pt">~Chuck Bredestege</p></div></div>