Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.
Hello,
I've recently setup an InfiniBand 40Gbit interconnect between two nodes to
run DRBD on top of some pretty fast storage. I am able to get DRBD to work
over Ethernet and IPoIB, however, when I try to enable SDP for the lower
latency, lower overhead communication I'm getting connection errors:
block drbd0: conn( Unconnected -> WFConnection )
block drbd0: connect failed, err = -22
block drbd0: connect failed, err = -22
block drbd0: connect failed, err = -22
I have the MLNX_OFED installed on CentOS5.5 with SDP active:
# rpm -qa|grep sdp
libsdp-devel-1.1.100-0.1.g920ea31
sdpnetstat-1.60-0.2.g8844f04
libsdp-1.1.100-0.1.g920ea31
libsdp-1.1.100-0.1.g920ea31
libsdp-devel-1.1.100-0.1.g920ea31
libsdp-debuginfo-1.1.100-0.1.g920ea31
[root at node02 log]# netperf -f g -H 192.168.20.1 -c -C
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.20.1
(192.168.20.1) port 0 AF_INET
Recv Send Send Utilization Service
Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local
remote
bytes bytes bytes secs. 10^9bits/s % S % S us/KB us/KB
87380 65536 65536 10.00 16.15 1.74 4.61 0.211 0.562
[root at node02 log]# LD_PRELOAD="libsdp.so" netperf -f g -H 192.168.20.1 -c -C
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.20.1
(192.168.20.1) port 0 AF_INET
Recv Send Send Utilization Service
Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local
remote
bytes bytes bytes secs. 10^9bits/s % S % S us/KB us/KB
87380 65536 65536 10.01 24.67 3.18 3.28 0.253 0.262
There is a significant (50-100%) increase in bandwidth and decrease in
latency using SDP instead of IPoIB, so even though IPoIB works I'd like to
use the SDP method.
Here's my node stanzas:
<snip>
# 90-subnet = Ethernet, 20-subnet = InfiniBand
on node01 {
address 192.168.20.1:7778;
# address sdp 192.168.20.1:7778;
# address 192.168.90.1:7778;
}
on node02 {
address 192.168.20.2:7778;
# address sdp 192.168.20.2:7778;
# address 192.168.90.2:7778;
}
</snip>
If I uncomment on the SDP-based address, I get connection error -22 and
haven't been able to surmise what that means. Any ideas? Running the
lastest DRBD:
# rpm -qa|grep drbd
drbd83-8.3.8.1-1
kmod-drbd83-8.3.8.1-1
Any help would be appreciated,
-JR
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20100916/7a31a7fe/attachment.htm>