Hello,<div><br></div><div>I've recently setup an InfiniBand 40Gbit interconnect between two nodes to run DRBD on top of some pretty fast storage. I am able to get DRBD to work over Ethernet and IPoIB, however, when I try to enable SDP for the lower latency, lower overhead communication I'm getting connection errors:</div>
<div><br></div><div><div><div>block drbd0: conn( Unconnected -> WFConnection )</div><div>block drbd0: connect failed, err = -22</div><div>block drbd0: connect failed, err = -22</div><div>block drbd0: connect failed, err = -22</div>
</div><div><br></div><div>I have the MLNX_OFED installed on CentOS5.5 with SDP active:</div><div><br></div><div># rpm -qa|grep sdp</div><div>libsdp-devel-1.1.100-0.1.g920ea31</div><div>sdpnetstat-1.60-0.2.g8844f04</div><div>
libsdp-1.1.100-0.1.g920ea31</div><div>libsdp-1.1.100-0.1.g920ea31</div><div>libsdp-devel-1.1.100-0.1.g920ea31</div><div>libsdp-debuginfo-1.1.100-0.1.g920ea31</div></div><div><br></div><div><div>[root@node02 log]# netperf -f g -H 192.168.20.1 -c -C</div>
<div>TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.20.1 (192.168.20.1) port 0 AF_INET</div><div>Recv Send Send Utilization Service Demand</div><div>Socket Socket Message Elapsed Send Recv Send Recv</div>
<div>Size Size Size Time Throughput local remote local remote</div><div>bytes bytes bytes secs. 10^9bits/s % S % S us/KB us/KB</div><div><br></div><div> 87380 65536 65536 10.00 16.15 1.74 4.61 0.211 0.562</div>
</div><div><br></div><div><div>[root@node02 log]# LD_PRELOAD="libsdp.so" netperf -f g -H 192.168.20.1 -c -C</div><div>TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.20.1 (192.168.20.1) port 0 AF_INET</div>
<div>Recv Send Send Utilization Service Demand</div><div>Socket Socket Message Elapsed Send Recv Send Recv</div><div>Size Size Size Time Throughput local remote local remote</div>
<div>bytes bytes bytes secs. 10^9bits/s % S % S us/KB us/KB</div><div><br></div><div> 87380 65536 65536 10.01 24.67 3.18 3.28 0.253 0.262</div></div><div><br></div><div>There is a significant (50-100%) increase in bandwidth and decrease in latency using SDP instead of IPoIB, so even though IPoIB works I'd like to use the SDP method.</div>
<div><br></div><div>Here's my node stanzas:</div><div><br></div><div><snip></div><div> # 90-subnet = Ethernet, 20-subnet = InfiniBand</div><div><div> on node01 {</div><div> address <a href="http://192.168.20.1:7778">192.168.20.1:7778</a>;</div>
<div># address sdp <a href="http://192.168.20.1:7778">192.168.20.1:7778</a>;</div><div># address <a href="http://192.168.90.1:7778">192.168.90.1:7778</a>;</div><div> }</div><div> on node02 {</div><div> address <a href="http://192.168.20.2:7778">192.168.20.2:7778</a>;</div>
<div># address sdp <a href="http://192.168.20.2:7778">192.168.20.2:7778</a>;</div><div># address <a href="http://192.168.90.2:7778">192.168.90.2:7778</a>;</div><div> }</div></div><div></snip></div><div><br></div>
<div>If I uncomment on the SDP-based address, I get connection error -22 and haven't been able to surmise what that means. Any ideas? Running the lastest DRBD:</div><div><br></div><div><div># rpm -qa|grep drbd</div>
<div>drbd83-8.3.8.1-1</div><div>kmod-drbd83-8.3.8.1-1</div></div><div><br></div><div>Any help would be appreciated,</div><div>-JR</div>