<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#ffffff">
Hi,<br>
<br>
Em 01-06-2010 08:16, Robert Dunkley escreveu:
<blockquote
cite="mid:C1EAC9C5E752D24C968FF091D446D82360D2FF@ALTERNATEREALIT"
type="cite">
<meta http-equiv="Content-Type"
content="text/html; charset=ISO-8859-1">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="Section1">
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Hi
Igor,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">SDP
performance can be measured with the Qperf utility included
with the OFED drivers/utilities. I was testing with single port 3<sup>rd</sup>
Gen (Same series as the ones you link) Mellanox DDR (20/16Gbit) cards
on
Opterons 2 years ago and was getting 11Gbit/sec. The newer 4<sup>th</sup>
gen
ConnectX mellanox cards will achieve 13-14Gbit/sec in DDR so if you are
buying
new get ConnectX.</span></p>
</div>
</blockquote>
<br>
I understand what you are trying to tell, but ConnectX it's so much
expensive then the one I mention and the bandwith provided in Mellanox
MHES14-xtc will be more then enought for our setup's.<br>
<span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"><o:p><br>
</o:p></span>
<blockquote
cite="mid:C1EAC9C5E752D24C968FF091D446D82360D2FF@ALTERNATEREALIT"
type="cite">
<div class="Section1">
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">For
comparison in datagram mode I was getting 3.2Gbit/sec and in
connected mode with a 32Kb MTU in connected mode I was getting
7Gbit/sec. This
was 2 years ago so new version of OFED may well have improved
performance.</span></p>
</div>
</blockquote>
<br>
I will have to try myself, I do not know the technology. I'm just
trying to make sure I will not buy anything that don't work out and
stuck with the cards in the closet.<br>
<br>
Thanks very much for your help.<br>
<br>
<blockquote
cite="mid:C1EAC9C5E752D24C968FF091D446D82360D2FF@ALTERNATEREALIT"
type="cite">
<div class="Section1">
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Hope
this helps,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Rob<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<div>
<div
style="border-style: solid none none; border-color: rgb(181, 196, 223) -moz-use-text-color -moz-use-text-color; border-width: 1pt medium medium; padding: 3pt 0cm 0cm;">
<p class="MsoNormal"><b><span
style="font-size: 10pt; font-family: "Tahoma","sans-serif"; color: windowtext;"
lang="EN-US">From:</span></b><span
style="font-size: 10pt; font-family: "Tahoma","sans-serif"; color: windowtext;"
lang="EN-US">
<a class="moz-txt-link-abbreviated" href="mailto:drbd-user-bounces@lists.linbit.com">drbd-user-bounces@lists.linbit.com</a>
[<a class="moz-txt-link-freetext" href="mailto:drbd-user-bounces@lists.linbit.com">mailto:drbd-user-bounces@lists.linbit.com</a>] <b>On
Behalf Of </b>Igor Neves<br>
<b>Sent:</b> 31 May 2010 14:33<br>
<b>To:</b> Michael Iverson<br>
<b>Cc:</b> <a class="moz-txt-link-abbreviated" href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br>
<b>Subject:</b> Re: [DRBD-user] Infiniband card support and help<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi,<br>
<br>
<br>
<br>
On 05/31/2010 12:45 PM, Michael Iverson wrote: <o:p></o:p></p>
<div>
<p class="MsoNormal">Igor, <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I'm basically doing the same thing, only with
MHEA28-XTC
cards. I wouldn't think you'll have any problems creating a similar
setup with
the MHES cards.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I've not attempted to use infiniband sdr, just
ipoib. I am
running opensm on one of the nodes. I'm getting throughput numbers like
this:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><br>
Would be very nice if you could test your setup with drbd infiniband
sdr
support, probably you will not need to re-sync anything.<br>
<br>
<br>
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">cirrus:~$ netperf -H stratus-ib<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0
AF_INET to <a moz-do-not-send="true"
href="http://stratus-ib.focus1.com">stratus-ib.focus1.com</a>
(172.16.24.1)
port 0 AF_INET : demo<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Recv Send Send
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Socket Socket Message Elapsed
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Size Size Size Time
Throughput <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">bytes bytes bytes secs.
10^6bits/sec <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> 87380 16384 16384 10.00
7861.61 <o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><br>
Nice.<br>
<br>
<br>
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">A couple of things to watch out for:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">1. Upgrade the firmware on the cards to the
latest and
greatest version. I saw about a 25% increase in throughput as a result.
The
firmware updater was a pain to compile, but that was mostly due to
Ubuntu's
fairly rigid default compiler flags.<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
Will watch that!<br>
<br>
<br>
<o:p></o:p></p>
<div>
<p class="MsoNormal">2. Run the cards in connected mode, rather than
datagram
mode, and put the MTU at the max value of 65520. My performance
benchmarks of
drbd show that this is the best setup. <o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
If I use infiniband sdr support from drbd, should I care about MTU?<br>
<br>
<br>
<o:p></o:p></p>
<div>
<p class="MsoNormal">The replication rate on my setup is completely
limited by
the bandwidth of my disk subsystem, which is about 200 MB/s for writes.
I can
share some performance comparisons between this and bonded gigabit
ethernet, if
you would like. However, I won't be able to provide it until tomorrow,
as it is
a holiday in the US today, and I don't have ready access to the data.<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
We have a couple of setups that have I/O performances greater than
500MB/sec,
so we really need 10Gbit trunks. <br>
<br>
Thanks for the help, but I don't need performance results from Gbit
setup's, we
have a couple, and we know the problems! :) Anyway if you want to paste
it
here, I guess no one will complain.<br>
<br>
<br>
<o:p></o:p></p>
<div>
<p class="MsoNormal">On Mon, May 31, 2010 at 6:17 AM, Igor Neves <<a
moz-do-not-send="true" href="mailto:igor@3gnt.net">igor@3gnt.net</a>>
wrote:<o:p></o:p></p>
<p class="MsoNormal">Hi,<br>
<br>
I'm looking for a 10Gbit backend for storage drbd replication. I'm
expecting to
setup infiniband solution connected back to back, this means both nodes
will be
connected together without a switch.<br>
<br>
I wonder if I bought two of this cards MHES14-xtc and a cable, I will
be able
to produce such setup?<br>
<br>
Link to the cards: <a moz-do-not-send="true"
href="http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=19&menu_section=41"
target="_blank">http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=19&menu_section=41</a><br>
<br>
Another question, I intend to use this with infiniband sdr support
added to
drbd in 8.3.3, and I found this on the spec's of the card.<br>
<br>
"In addition, the card includes internal Subnet Management Agent (SMA)
and
General Service Agents, eliminating the requirement for an external
management
agent CPU."<br>
<br>
This means I don't need to run openSM in any nodes? I will just need to
setup
two cards, a cable, connect them, and setup IPoIB to start replicating
in
10Gbit?<br>
<br>
Thanks very much,<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
Thanks, once again.<br>
<br>
<o:p></o:p></p>
<pre>-- <o:p></o:p></pre>
<pre>Igor Neves <a moz-do-not-send="true"
href="mailto:igor.neves@3gnt.net"><igor.neves@3gnt.net></a><o:p></o:p></pre>
<pre>3GNTW - Tecnologias de Informação, Lda<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre> SIP: <a moz-do-not-send="true" href="mailto:igor@3gnt.net">igor@3gnt.net</a><o:p></o:p></pre>
<pre> MSN: <a moz-do-not-send="true" href="mailto:igor@3gnt.net">igor@3gnt.net</a><o:p></o:p></pre>
<pre> JID: <a moz-do-not-send="true" href="mailto:igor@3gnt.net">igor@3gnt.net</a><o:p></o:p></pre>
<pre> PSTN: 00351 252377120<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
</div>
<!--[object_id=#saq.co.uk#]--><font face="Tahoma"><font color="#0000ff">
<p class="MsoPlainText" style="margin: 0cm 0cm 0pt;"><span
style="font-size: 10pt; color: black; font-family: 'Verdana','sans-serif';"><strong></strong></span></p>
<p class="MsoPlainText" style="margin: 0cm 0cm 0pt;"><font size="2"><span
style="font-size: 10pt; color: black; font-family: 'Verdana','sans-serif';"><strong>The
SAQ Group</strong></span></font></p>
<p class="MsoPlainText" style="margin: 0cm 0cm 0pt;"><span
style="font-size: 10pt; color: black; font-family: 'Verdana','sans-serif';"><strong>Registered
Office: <st1:street w:st="on"><st1:address w:st="on">18 Chapel Street,
Petersfield, Hampshire GU32 3DZ</st1:address></st1:street></strong><br>
SAQ is the trading name of SEMTEC Limited. Registered in <st1:country-region
w:st="on">England</st1:country-region> & <st1:country-region
w:st="on"><st1:place w:st="on">Wales</st1:place></st1:country-region><br>
Company Number: 06481952</span></p>
<p class="MsoPlainText" style="margin: 0cm 0cm 0pt;"><span
style="font-size: 10pt; color: black; font-family: 'Verdana','sans-serif';"><o:p> </o:p></span></p>
<p class="MsoPlainText" style="margin: 0cm 0cm 0pt;"><span
style="font-size: 10pt; color: black; font-family: 'Verdana','sans-serif';"><a
moz-do-not-send="true" href="http://www.saqnet.co.uk/">http://www.saqnet.co.uk</a>
AS29219<o:p></o:p></span></p>
<p class="MsoPlainText" style="margin: 0cm 0cm 0pt;"><span
style="font-size: 10pt; color: black; font-family: 'Verdana','sans-serif';">SAQ
Group Delivers high quality, honestly priced communication and I.T.
services to UK Business.<o:p></o:p></span></p>
<p class="MsoPlainText" style="margin: 0cm 0cm 0pt;"><span
style="font-size: 10pt; color: black; font-family: 'Verdana','sans-serif';">Broadband :
Domains : Email : Hosting : CoLo : Servers : Racks : Transit : Backups
: Managed Networks : Remote Support.</span></p>
<p class="MsoPlainText" style="margin: 0cm 0cm 0pt;"><span
style="font-size: 10pt; color: black; font-family: 'Verdana','sans-serif';"></span><font
size="2"> </font></p>
<p class="MsoPlainText" style="margin: 0cm 0cm 0pt;"><font size="2"> <img
moz-do-not-send="true" alt="SAQ Group"
src="http://www.saq.co.uk/office/saqlogo.gif" hspace="0"
align="baseline" border="0"></font></p>
<p class="MsoPlainText" style="margin: 0cm 0cm 0pt;"> </p>
<p class="MsoPlainText" style="margin: 0cm 0cm 0pt;"><font size="2"><font
color="#000000"><strong>ISPA Member</strong></font></font></p>
</font></font></blockquote>
<br>
<pre class="moz-signature" cols="72">--
Igor Neves <a class="moz-txt-link-rfc2396E" href="mailto:igor.neves@3gnt.net"><igor.neves@3gnt.net></a>
3GNTW - Tecnologias de Informação, Lda
SIP: <a class="moz-txt-link-abbreviated" href="mailto:igor@3gnt.net">igor@3gnt.net</a>
MSN: <a class="moz-txt-link-abbreviated" href="mailto:igor@3gnt.net">igor@3gnt.net</a>
JID: <a class="moz-txt-link-abbreviated" href="mailto:igor@3gnt.net">igor@3gnt.net</a>
PSTN: 00351 252377120
</pre>
</body>
</html>