Hello Lars =)<div><br></div><div>Reply inline:<br><br><div class="gmail_quote">On Fri, Sep 17, 2010 at 4:22 AM, Lars Ellenberg <span dir="ltr"><<a href="mailto:lars.ellenberg@linbit.com">lars.ellenberg@linbit.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im"><br>
</div>-EINVAL<br>
<br>
iirc, it is a bug inside the in-kernel SDP connect() peer lookup,<br>
which EINVALs if the target address is not given as AF_INET (!),<br>
even if the socket itself is AF_INET_SDP.<br>
Or the other way around.<br></blockquote><div><br></div><div>I see, great info.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
If you do "drbdadm -d connect $resource", you get the drbdsetup<br>
command that would have been issued.<br>
replace the second (remmote) sdp with ipv4,<br>
and do them manually, on both nodes.<br>
If that does not work, replace only the first (local) sdp with ipv4,<br>
but keep the second (remote) sdp.<br></blockquote><div><br></div><div><div>The first suggestion seems to work:</div><div><br></div><div><div>[root@node01 ~]# drbdsetup 0 net sdp:<a href="http://192.168.20.1:7778">192.168.20.1:7778</a> ipv4:<a href="http://192.168.20.2:7778">192.168.20.2:7778</a> C --set-defaults --create-device --max-epoch-size=20000 --max-buffers=20000 --after-sb-2pri=disconnect --after-sb-1pri=discard-secondary --after-sb-0pri=discard-zero-changes --allow-two-primaries</div>
</div><div><br></div><div><div>[root@node02 ~]# drbdsetup 0 net sdp:<a href="http://192.168.20.2:7778">192.168.20.2:7778</a> ipv4:<a href="http://192.168.20.1:7778">192.168.20.1:7778</a> C --set-defaults --create-device --max-epoch-size=20000 --max-buffers=20000 --after-sb-2pri=disconnect --after-sb-1pri=discard-secondary --after-sb-0pri=discard-zero-changes --allow-two-primaries</div>
</div></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
If that gets you connected, then its that bug.<br>
I think I even patched it in kernel once,<br>
but don't find that right now,<br>
and don't remember the SDP version either.<br>
I think it was<br>
drivers/infiniband/ulp/sdp/sdp_main.c:addr_resolve_remote()<br>
missing an (... || ... = AF_INET_SDP)<br></blockquote><div><br></div><div>Hmmm. It may be a MLNX_OFED specific bug? If we can get the code, I have a support path with Mellanox, I can probably get this pushed into their upstream OFED. I'll look at it and see if I can figure it out.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im"><br>That's all userland, and does not affect DRBD, as DRBD does all</div>
networking from within the kernel.<br></blockquote><div><br></div><div>Ahhh, right.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Share your findings on DRBD performance IPoIB vs. SDP,<br>
once you get the thing to work on your platform.</blockquote><div><br></div><div>Well, using the drbdsetup method above, SDP is significantly slower than IPoIB. Below are some write results, the HAStorage VG is using drbd0 as a PV:</div>
<div><br></div><div>With IPoIB:</div><div><br></div><div><p class="MsoNormal"># time dd if=/dev/zero of=/dev/HAStorage/test bs=4096M count=5</p>
<p class="MsoNormal">0+5 records in</p>
<p class="MsoNormal">0+5 records out</p>
<p class="MsoNormal">10737397760 bytes (11 GB) copied, 17.8384 seconds, 602 MB/s</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">real 0m17.864s</p>
<p class="MsoNormal">user 0m0.000s</p>
<p class="MsoNormal">sys 0m10.292s</p></div><div><br></div><div>With SDP and the drbdsetup sdp/ipv4:</div><div><br></div><div><div># time dd if=/dev/zero of=/dev/HAStorage/test bs=4096M count=5</div><div>0+5 records in</div>
<div>0+5 records out</div><div>10737397760 bytes (11 GB) copied, 26.2015 seconds, 410 MB/s</div><div><br></div><div>real 0m26.220s</div><div>user 0m0.001s</div><div>sys 0m11.891s</div></div><div><br></div><div><br>
</div><div>The underlying storage is a 16-disk RAID10 with 1.5GB of flash-backed write cache. Local writes to storage happen around 900MB/s sustained, and burst at several GB/sec to write cache. I suspect a proper fix for the in-kernel SDP connect() may fix SDP performance here. At least I would hope so!</div>
<div><br></div><div>Thanks for the information, very helpful,</div><div>-JR</div></div></div>