<p dir="ltr"><br>
On 29/02/2016 2:00 AM, "翟果" <<a href="mailto:zhai.guo@datatom.com">zhai.guo@datatom.com</a>> wrote:<br>
><br>
> Hello,All:<br>
> I used to google for the solution,but get no answers.<br>
> Somebody says DRBD8.4 doesn't work with sdp?Really?<br>
As far as I can see thats not true: <a href="http://drbd.linbit.com/users-guide-8.4/s-replication-transports.html">http://drbd.linbit.com/users-guide-8.4/s-replication-transports.html</a></p>
<p dir="ltr">> I have two nodes(Centos 6.4 with kernel 2.6.32-358.el6.x86_64),and two Mellanox IB cards.Now,I want to use DRBD to sync data between two nodes.But the DRBD status is always "WFconnect".<br>
><br>
> [root@node1 home]# cat /proc/drbd <br>
> version: 8.4.2 (api:1/proto:86-101)<br>
> GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by root@localhost.localdomain, 2013-09-30 16:29:29<br>
> 0: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/Outdated C r----s<br>
> ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:10485404<br>
><br>
><br>
> I have loaded the ib_sdp.ko and pre_loaded lib_sdp.so<br>
><br>
> [root@node36 ofa_kernel-3.1]# lsmod | grep sdp<br>
> ib_sdp 129946 0 <br>
> rdma_cm 43237 2 rdma_ucm,ib_sdp<br>
> ib_core 126865 13 rdma_ucm,ib_ucm,ib_sdp,rdma_cm,iw_cm,ib_ipoib,ib_cm,ib_uverbs,ib_umad,mlx5_ib,mlx4_ib,ib_sa,ib_mad<br>
> mlx_compat 32626 18 rdma_ucm,ib_ucm,ib_sdp,rdma_cm,iw_cm,ib_ipoib,ib_cm,ib_uverbs,ib_umad,mlx5_ib,mlx5_core,mlx4_en,mlx4_ib,ib_sa,ib_mad,ib_core,ib_addr,mlx4_core<br>
> ipv6 321422 41 ib_sdp,ib_ipoib,ib_core,ib_addr,ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6<br>
><br>
> [root@node1 ofa_kernel-3.1]# export<br>
> declare -x G_BROKEN_FILENAMES="1"<br>
> declare -x HISTCONTROL="ignoredups"<br>
> declare -x HISTSIZE="1000"<br>
> declare -x HOME="/root"<br>
> declare -x HOSTNAME="node1"<br>
> declare -x LANG="en_US.UTF-8"<br>
> declare -x LD_PRELOAD="/usr/lib64/libsdp.so"<br>
> declare -x LESSOPEN="|/usr/bin/lesspipe.sh %s"<br>
> declare -x LIBSDP_CONFIG_FILE="/etc/libsdp.conf"<br>
> declare -x LOGNAME="root"<br>
><br>
> Here is part of my configure:<br>
> resource res_sdp {<br>
> meta-disk internal;<br>
> disk /dev/mapper/p1-lv1;<br>
> device /dev/drbd0;<br>
><br>
> on node1 {<br>
> address sdp *****:7789;<br>
> }<br>
> on node36 {<br>
> address sdp *****:7789;<br>
> }<br>
> }<br>
><br>
> And I get some log of ib_sdp.ko.<br>
><br>
> Feb 28 15:24:20 node1 kernel: sdp_cma_handler:657 sdp_sock( 1465:0 58262:7789): event: RDMA_CM_EVENT_CONNECT_RESPONSE handled<br>
> Feb 28 15:24:20 node1 kernel: sdp_cma_handler:671 sdp_sock( 1465:0 58262:7789): event: RDMA_CM_EVENT_CONNECT_RESPONSE done. status 0<br>
> Feb 28 15:24:27 node1 kernel: sdp_cma_handler:438 sdp_sock( 1465:0 7789:0): event: RDMA_CM_EVENT_CONNECT_REQUEST<br>
> Feb 28 15:24:27 node1 kernel: sdp_connect_handler:178 sdp_sock( 1465:0 7789:0): sdp_connect_handler ffff88086ee7ac00 -> ffff88086ef41000<br>
> Feb 28 15:24:27 node1 kernel: sdp_init_sock:1325 sdp_sock( 1465:0 7789:0): sdp_init_sock<br>
> Feb 28 15:24:27 node1 kernel: sdp_init_qp:111 sdp_sock( 1465:0 7789:41102): sdp_init_qp<br>
> Feb 28 15:24:27 node1 kernel: sdp_init_qp:114 sdp_sock( 1465:0 7789:41102): Max sges: 32<br>
> Feb 28 15:24:27 node1 kernel: sdp_init_qp:117 sdp_sock( 1465:0 7789:41102): Setting max send sge to: 9<br>
> Feb 28 15:24:27 node1 kernel: sdp_init_qp:120 sdp_sock( 1465:0 7789:41102): Setting max recv sge to: 9<br>
> Feb 28 15:24:27 node1 kernel: sdp_init_qp:151 sdp_sock( 1465:0 7789:41102): sdp_init_qp done<br>
> Feb 28 15:24:27 node1 kernel: _sdp_exch_state:559 sdp_sock( 1465:0 7789:41102): sdp_connect_handler:300 - set state: TCP_LISTEN -> TCP_SYN_RECV 0x480<br>
> Feb 28 15:24:27 node1 kernel: sdp_cma_handler:657 sdp_sock( 1465:0 7789:0): event: RDMA_CM_EVENT_CONNECT_REQUEST handled<br>
> Feb 28 15:24:27 node1 kernel: sdp_cma_handler:671 sdp_sock( 1465:0 7789:0): event: RDMA_CM_EVENT_CONNECT_REQUEST done. status 0<br>
> Feb 28 15:24:28 node1 kernel: sdp_cma_handler:438 sdp_sock( 1465:0 58262:7789): event: RDMA_CM_EVENT_DISCONNECTED<br>
> Feb 28 15:24:28 node1 kernel: _sdp_exch_state:559 sdp_sock( 1465:0 58262:7789): sdp_set_error:591 - set state: TCP_ESTABLISHED -> TCP_CLOSE 0xffffffff<br>
> Feb 28 15:24:28 node1 kernel: sdp_disconnected_handler:400 sdp_sock( 1465:0 58262:7789): sdp_disconnected_handler<br>
> Feb 28 15:24:28 node1 kernel: sdp_cma_handler:657 sdp_sock( 1465:0 58262:7789): event: RDMA_CM_EVENT_DISCONNECTED handled<br>
> Feb 28 15:24:28 node1 kernel: sdp_reset_sk:492 sdp_sock( 1465:0 58262:7789): sdp_reset_sk<br>
> Feb 28 15:24:28 node1 kernel: sdp_reset_sk:501 sdp_sock( 1465:0 58262:7789): setting state to error<br>
> Feb 28 15:24:28 node1 kernel: _sdp_exch_state:559 sdp_sock( 1465:0 58262:7789): sdp_set_error:591 - set state: TCP_CLOSE -> TCP_CLOSE 0xffffffff<br>
> Feb 28 15:24:28 node1 kernel: sdp_cma_handler:671 sdp_sock( 1465:0 58262:7789): event: RDMA_CM_EVENT_DISCONNECTED done. status -104<br>
> Feb 28 15:24:28 node1 kernel: sdp_destroy_work:1238 sdp_sock( 1615:6 58262:7789): sdp_destroy_work: refcnt 2<br>
> Feb 28 15:24:28 node1 kernel: sdp_do_posts:816 sdp_sock( 1614:11 58262:7789): QP is deactivated<br>
> Feb 28 15:24:28 node1 kernel: sdp_do_posts:816 sdp_sock( 1614:11 58262:7789): QP is deactivated<br>
> Feb 28 15:24:28 node1 kernel: sdp_destroy_qp:242 sdp_sock( 1615:6 58262:7789): destroying qp<br>
> Feb 28 15:24:28 node1 kernel: _sdp_exch_state:559 sdp_sock( 1615:0 58262:7789): sdp_destroy_work:1264 - set state: TCP_CLOSE -> TCP_CLOSE 0xffffffff<br>
> Feb 28 15:24:31 node1 kernel: sdp_cma_handler:438 sdp_sock( 1465:0 7789:0): event: RDMA_CM_EVENT_CONNECT_REQUEST<br>
> Feb 28 15:24:31 node1 kernel: sdp_connect_handler:178 sdp_sock( 1465:0 7789:0): sdp_connect_handler ffff88086ee7ac00 -> ffff88086dd43000<br>
> Feb 28 15:24:31 node1 kernel: sdp_init_sock:1325 sdp_sock( 1465:0 7789:0): sdp_init_sock<br>
> Feb 28 15:24:31 node1 kernel: sdp_init_qp:111 sdp_sock( 1465:0 7789:40747): sdp_init_qp<br>
> Feb 28 15:24:31 node1 kernel: sdp_init_qp:114 sdp_sock( 1465:0 7789:40747): Max sges: 32<br>
> Feb 28 15:24:31 node1 kernel: sdp_init_qp:117 sdp_sock( 1465:0 7789:40747): Setting max send sge to: 9<br>
> Feb 28 15:24:31 node1 kernel: sdp_init_qp:120 sdp_sock( 1465:0 7789:40747): Setting max recv sge to: 9<br>
> Feb 28 15:24:31 node1 kernel: sdp_init_qp:151 sdp_sock( 1465:0 7789:40747): sdp_init_qp done<br>
> Feb 28 15:24:31 node1 kernel: _sdp_exch_state:559 sdp_sock( 1465:0 7789:40747): sdp_connect_handler:300 - set state: TCP_LISTEN -> TCP_SYN_RECV 0x480<br>
> Feb 28 15:24:31 node1 kernel: sdp_cma_handler:657 sdp_sock( 1465:0 7789:0): event: RDMA_CM_EVENT_CONNECT_REQUEST handled<br>
> Feb 28 15:24:31 node1 kernel: sdp_cma_handler:671 sdp_sock( 1465:0 7789:0): event: RDMA_CM_EVENT_CONNECT_REQUEST done. status 0<br>
> Feb 28 15:24:31 node1 kernel: sdp_cma_handler:438 sdp_sock( 1465:0 7789:40747): event: RDMA_CM_EVENT_ESTABLISHED<br>
> Feb 28 15:24:31 node1 kernel: sdp_connected_handler:358 sdp_sock( 1465:0 7789:40747): sdp_connected_handler<br>
> Feb 28 15:24:31 node1 kernel: _sdp_exch_state:559 sdp_sock( 1465:0 7789:40747): sdp_connected_handler:363 - set state: TCP_SYN_RECV -> TCP_ESTABLISHED 0x8<br>
> Feb 28 15:24:31 node1 kernel: sdp_set_default_moderation:361 sdp_sock( 1465:0 7789:40747): Using adaptive interrupt moderation<br>
> Feb 28 15:24:31 node1 kernel: sdp_set_default_moderation:374 sdp_sock( 1465:0 7789:40747): Default coalesing params for buf size:32784 - moder_cnt:10 moder_time:16<br>
> Feb 28 15:24:31 node1 kernel: sdp_connected_handler:383 sdp_sock( 1465:0 7789:0): sdp_connected_handler child connection established<br>
> Feb 28 15:24:31 node1 kernel: sdp_cma_handler:657 sdp_sock( 1465:0 7789:40747): event: RDMA_CM_EVENT_ESTABLISHED handled<br>
> Feb 28 15:24:31 node1 kernel: sdp_cma_handler:671 sdp_sock( 1465:0 7789:40747): event: RDMA_CM_EVENT_ESTABLISHED done. status 0<br>
> Feb 28 15:24:32 node1 kernel: sdp_do_posts:816 sdp_sock( 3662:11 58262:7789): QP is deactivated<br>
> Feb 28 15:24:32 node1 kernel: sdp_close:751 sdp_sock( 3662:11 58262:7789): sdp_close<br>
> Feb 28 15:24:32 node1 kernel: sdp_delete_keepalive_timer:275 sdp_sock( 3662:11 58262:7789): sdp_delete_keepalive_timer<br>
> Feb 28 15:24:32 node1 kernel: sdp_close:834 sdp_sock( 3662:11 58262:7789): sdp_close:834 - sock_put(SOCK_REF_ALIVE) - refcount = 1 from withing sk_common_release<br>
> Feb 28 15:24:32 node1 kernel: sdp_unhash:2671 sdp_sock( 3662:11 58262:7789): sdp_unhash<br>
> Feb 28 15:24:32 node1 kernel: sdp_destruct:596 sdp_sock( 3662:11 58262:7789): sdp_destruct<br>
> Feb 28 15:24:32 node1 kernel: sdp_destroy_resources:537 sdp_sock( 3662:11 58262:7789): sdp_destroy_resources<br>
> Feb 28 15:24:32 node1 kernel: sdp_destroy_qp:242 sdp_sock( 3662:11 58262:7789): destroying qp<br>
> Feb 28 15:24:32 node1 kernel: sdp_destroy_resources:572 sdp_sock( 3662:11 58262:7789): sdp_destroy_resources done; releasing sock<br>
> Feb 28 15:24:32 node1 kernel: sdp_destruct:648 sdp_sock( 3662:11 58262:7789): sdp_destruct done<br>
> Feb 28 15:24:32 node1 kernel: sdp_create_ipvx_socket:2838 sdp_sock( 3662:11 -1:-1): type 1 protocol 6<br>
> Feb 28 15:24:32 node1 kernel: sdp_init_sock:1325 sdp_sock( 3662:11 0:0): sdp_init_sock<br>
> Feb 28 15:24:32 node1 kernel: sdp_get_port:216 sdp_sock( 3662:11 0:0): sdp_get_port: <a href="http://192.168.1.31:0">192.168.1.31:0</a><br>
> Feb 28 15:24:32 node1 kernel: sdp_ipv4_connect:964 sdp_sock( 3662:11 59104:0): sdp_ipv4_connect <a href="http://192.168.1.31:57574">192.168.1.31:57574</a> -> <a href="http://192.168.1.36:7789">192.168.1.36:7789</a><br>
> Feb 28 15:24:32 node1 kernel: _sdp_exch_state:559 sdp_sock( 3662:11 59104:0): sdp_connect:1003 - set state: TCP_CLOSE -> TCP_SYN_SENT 0x80<br>
> Feb 28 15:24:32 node1 kernel: sdp_cma_handler:438 sdp_sock( 1009:1 59104:0): event: RDMA_CM_EVENT_ADDR_RESOLVED<br>
> Feb 28 15:24:32 node1 kernel: sdp_cma_handler:657 sdp_sock( 1009:1 59104:0): event: RDMA_CM_EVENT_ADDR_RESOLVED handled<br>
> Feb 28 15:24:32 node1 kernel: sdp_cma_handler:671 sdp_sock( 1009:1 59104:0): event: RDMA_CM_EVENT_ADDR_RESOLVED done. status 0<br>
> Feb 28 15:24:32 node1 kernel: sdp_cma_handler:438 sdp_sock( 1599:6 59104:0): event: RDMA_CM_EVENT_ROUTE_RESOLVED<br>
><br>
> And what did the sdpnetstat say?<br>
> [root@node1 home]# sdpnetstat -S<br>
> Active Internet connections (w/o servers)<br>
> Proto Recv-Q Send-Q Local Address Foreign Address State <br>
> sdp 0 0 node1:office-tools node36:59756 ESTABLISHED <br>
> sdp 0 0 node1:office-tools node36:52031 ESTABLISHED <br>
> sdp 0 0 node1:office-tools node36:52658 ESTABLISHED <br>
> sdp 0 0 node1:54297 node36:office-tools ESTABLISHED <br>
><br>
> Any insight would be greatly appreciated.<br>
><br>
><br>
><br>
> _______________________________________________<br>
> drbd-user mailing list<br>
> <a href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br>
> <a href="http://lists.linbit.com/mailman/listinfo/drbd-user">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
><br>
</p>