<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#ffffff">
Hi,<br>
<br>
<br>
<br>
On 05/31/2010 12:45 PM, Michael Iverson wrote:
<blockquote
cite="mid:AANLkTikIQjR8PeV6zGKKlFrzUUc_XLbgvGojEdHSUM0m@mail.gmail.com"
type="cite">
<div>Igor, </div>
<div><br>
</div>
<div>I'm basically doing the same thing, only with MHEA28-XTC cards.
I wouldn't think you'll have any problems creating a similar setup with
the MHES cards.</div>
<div><br>
</div>
<div>I've not attempted to use infiniband sdr, just ipoib. I am
running opensm on one of the nodes. I'm getting throughput numbers like
this:</div>
<div><br>
</div>
</blockquote>
<br>
Would be very nice if you could test your setup with drbd infiniband
sdr support, probably you will not need to re-sync anything.<br>
<br>
<blockquote
cite="mid:AANLkTikIQjR8PeV6zGKKlFrzUUc_XLbgvGojEdHSUM0m@mail.gmail.com"
type="cite">
<div>
<div>cirrus:~$ netperf -H stratus-ib</div>
<div>TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to <a
moz-do-not-send="true" href="http://stratus-ib.focus1.com">stratus-ib.focus1.com</a>
(172.16.24.1) port 0 AF_INET : demo</div>
<div>Recv Send Send </div>
<div>Socket Socket Message Elapsed </div>
<div>Size Size Size Time Throughput </div>
<div>bytes bytes bytes secs. 10^6bits/sec </div>
<div><br>
</div>
<div> 87380 16384 16384 10.00 7861.61 <br>
</div>
</div>
</blockquote>
<br>
Nice.<br>
<br>
<blockquote
cite="mid:AANLkTikIQjR8PeV6zGKKlFrzUUc_XLbgvGojEdHSUM0m@mail.gmail.com"
type="cite">
<div><br>
</div>
<div>A couple of things to watch out for:</div>
<div><br>
</div>
<div>1. Upgrade the firmware on the cards to the latest and greatest
version. I saw about a 25% increase in throughput as a result. The
firmware updater was a pain to compile, but that was mostly due to
Ubuntu's fairly rigid default compiler flags.</div>
</blockquote>
<br>
Will watch that!<br>
<br>
<blockquote
cite="mid:AANLkTikIQjR8PeV6zGKKlFrzUUc_XLbgvGojEdHSUM0m@mail.gmail.com"
type="cite">
<div>2. Run the cards in connected mode, rather than datagram mode,
and put the MTU at the max value of 65520. My performance benchmarks of
drbd show that this is the best setup. <br>
</div>
</blockquote>
<br>
If I use infiniband sdr support from drbd, should I care about MTU?<br>
<br>
<blockquote
cite="mid:AANLkTikIQjR8PeV6zGKKlFrzUUc_XLbgvGojEdHSUM0m@mail.gmail.com"
type="cite">
<div>The replication rate on my setup is completely limited by the
bandwidth of my disk subsystem, which is about 200 MB/s for writes. I
can share some performance comparisons between this and bonded gigabit
ethernet, if you would like. However, I won't be able to provide it
until tomorrow, as it is a holiday in the US today, and I don't have
ready access to the data.</div>
</blockquote>
<br>
We have a couple of setups that have I/O performances greater than
500MB/sec, so we really need 10Gbit trunks. <br>
<br>
Thanks for the help, but I don't need performance results from Gbit
setup's, we have a couple, and we know the problems! :) Anyway if you
want to paste it here, I guess no one will complain.<br>
<br>
<blockquote
cite="mid:AANLkTikIQjR8PeV6zGKKlFrzUUc_XLbgvGojEdHSUM0m@mail.gmail.com"
type="cite">
<div class="gmail_quote">On Mon, May 31, 2010 at 6:17 AM, Igor Neves <span
dir="ltr"><<a moz-do-not-send="true" href="mailto:igor@3gnt.net">igor@3gnt.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hi,<br>
<br>
I'm looking for a 10Gbit backend for storage drbd replication. I'm
expecting to setup infiniband solution connected back to back, this
means both nodes will be connected together without a switch.<br>
<br>
I wonder if I bought two of this cards MHES14-xtc and a cable, I will
be able to produce such setup?<br>
<br>
Link to the cards: <a moz-do-not-send="true"
href="http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=19&menu_section=41"
target="_blank">http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=19&menu_section=41</a><br>
<br>
Another question, I intend to use this with infiniband sdr support
added to drbd in 8.3.3, and I found this on the spec's of the card.<br>
<br>
"In addition, the card includes internal Subnet Management Agent (SMA)
and General Service Agents, eliminating the requirement for an external
management agent CPU."<br>
<br>
This means I don't need to run openSM in any nodes? I will just need to
setup two cards, a cable, connect them, and setup IPoIB to start
replicating in 10Gbit?<br>
<br>
Thanks very much,<br>
</blockquote>
</div>
</blockquote>
<br>
Thanks, once again.<br>
<pre class="moz-signature" cols="72">--
Igor Neves <a class="moz-txt-link-rfc2396E" href="mailto:igor.neves@3gnt.net"><igor.neves@3gnt.net></a>
3GNTW - Tecnologias de Informação, Lda
SIP: <a class="moz-txt-link-abbreviated" href="mailto:igor@3gnt.net">igor@3gnt.net</a>
MSN: <a class="moz-txt-link-abbreviated" href="mailto:igor@3gnt.net">igor@3gnt.net</a>
JID: <a class="moz-txt-link-abbreviated" href="mailto:igor@3gnt.net">igor@3gnt.net</a>
PSTN: 00351 252377120
</pre>
</body>
</html>