<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns="http://www.w3.org/TR/REC-html40"
xmlns:ns1="http://schemas.microsoft.com/office/2004/12/omml">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--a:link
{mso-style-priority:99;}
span.MSOHYPERLINK
{mso-style-priority:99;}
a:visited
{mso-style-priority:99;}
span.MSOHYPERLINKFOLLOWED
{mso-style-priority:99;}
/* Font Definitions */
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:#606420;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:Arial;
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal;
font-family:Calibri;
color:#1F497D;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:Arial;
color:navy;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink="#606420">
<div class=Section1>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Hi Michael,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>This was quite interesting to read as it
isn’t that different from my setup. The raw device performance of
the raid on writes is about 550MB/s and reads at about 750MB/s. I haven’t
seen an 9260-8i with 1GB of RAM. Where did you find this card? I’m
using the LSI 9260-8i with 512MB.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>I’m also using Seagate Constellation
drives. However, the big difference between our machines is that you have
a Myricom card and I’ve got an Intel 82599 based 10Gbe card. On my
HP servers I always used the Myricom cards as they work extremely well but on
this order someone got sold on saving money using systems with the Intel LOM
setup. <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Have others had a lot of trouble using
Intel based NICs?<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Thanks,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Morey<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<div>
<div class=MsoNormal align=center style='text-align:center'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'>
<hr size=2 width="100%" align=center tabindex=-1>
</span></font></div>
<p class=MsoNormal><b><font size=2 face=Tahoma><span style='font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font size=2
face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'> Kushnir, Michael
(NIH/NLM/LHC) [C] [mailto:michael.kushnir@nih.gov] <br>
<b><span style='font-weight:bold'>Sent:</span></b> Thursday, September 29, 2011
4:14 PM<br>
<b><span style='font-weight:bold'>To:</span></b> Roof, Morey R.;
drbd-user@lists.linbit.com<br>
<b><span style='font-weight:bold'>Subject:</span></b> RE: performance help with
new servers</span></font><o:p></o:p></p>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#1f497d" face=Calibri><span
style='font-size:11.0pt;font-family:Calibri;color:#1F497D'>Hi Morey,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#1f497d" face=Calibri><span
style='font-size:11.0pt;font-family:Calibri;color:#1F497D'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#1f497d" face=Calibri><span
style='font-size:11.0pt;font-family:Calibri;color:#1F497D'>It really depends on
your RAID config and your network card. I am using Dell PE-C 2100s with
LSI’s </span></font><font size=2 face=Calibri><span
style='font-size:11.0pt;font-family:Calibri'>9260-8i RAID card with 1GB RAM on
board. I have write-back caching enabled and read-ahead caching as well. My RAID10
sets on each server is made up of 10 x 1TB Seagate Constellation 3Gb/s SATA
drives with 1MB stripe size. My 10GbE cards are Myricom SFP+ cards. On the
initial sync I was able to squeeze about 350MB/s transfer speeds between
servers. RAIDs 5 and 6 will give you worse performance. RAID 10, 50, and 60
should do much better. Newer LSI cards also have the option of adding SSDs as
read and/or write cache for your RAID array and various SSD protection features
to make them last. You will need the LSI GUI for Linux to find these features
easily. Also, I’ve heard of notoriously poor performance on 1<sup>st</sup>
gen Intel cards on Linux. <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>My setup is still pre-production so my drbd.conf file is
not finalized and has some non-advisable entries, but please take a look: <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>global {<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'># minor-count 64;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'># dialog-refresh 5; # 5 seconds<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'># disable-ip-verification;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>usage-count no;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>common {<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>syncer {<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> rate 100M; # My
bottleneck is RAID array so this is set so I can still use storage during sync.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> #al-extents
257;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> }<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>handlers {<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> #fence-peer
"/usr/lib64/heartbeat/drbd-peer-outdater -t 5";<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>
#pri-on-incon-degr "echo O > /proc/sysrq-trigger ; halt -f";<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>
#pri-lost-after-sb "echo O > /proc/sysrq-trigger ; halt -f";<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> #local-io-error
"echo O > /proc/sysrq-trigger ; halt -f";<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> }<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>disk {<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> fencing
resource-only;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>
no-disk-barrier;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> #use-bmbv;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> on-io-error
detach;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>
no-disk-flushes;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> }<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>net {<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> #
data-integrity-alg md5;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>
allow-two-primaries;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> after-sb-0pri
discard-older-primary;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> after-sb-1pri
discard-secondary;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> #after-sb-0pri
disconnect;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> #after-sb-1pri
disconnect;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> after-sb-2pri
disconnect;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> #rr-conflict
disconnect;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>startup {<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> wfc-timeout 5;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>
degr-wfc-timeout 5<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>
become-primary-on both;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> }<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>resource drbd0 {<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>protocol C;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>on **** {<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> device /dev/drbd0;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> disk /dev/sdb1;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> address ****;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> meta-disk internal;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>on ****{<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> device /dev/drbd0;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> disk /dev/sdb1;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> address ****;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> meta-disk internal;<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'> }<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>}<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>Best,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Calibri><span style='font-size:11.0pt;
font-family:Calibri'>Michael<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color="#1f497d" face=Calibri><span
style='font-size:11.0pt;font-family:Calibri;color:#1F497D'><o:p> </o:p></span></font></p>
<p class=MsoNormal><a name="_MailEndCompose"></a><font size=2 color="#1f497d"
face=Calibri><span style='font-size:11.0pt;font-family:Calibri;color:#1F497D'><o:p> </o:p></span></font></p>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><font size=2 face=Tahoma><span style='font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font size=2
face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'> Roof, Morey R.
[mailto:MRoof@admin.nmt.edu] <br>
<b><span style='font-weight:bold'>Sent:</span></b> Thursday, September 29, 2011
1:48 PM<br>
<b><span style='font-weight:bold'>To:</span></b> drbd-user@lists.linbit.com<br>
<b><span style='font-weight:bold'>Subject:</span></b> [DRBD-user] performance
help with new servers<o:p></o:p></span></font></p>
</div>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Hi Everyone,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>I got some new servers recently and they have the LSI
Megaraid based cards (with battery backup units) with 10K SAS drives and 10Gbe
cards. At the moment the performance is quite less than I would like with
them, about 90MB/s with DRBD protocol C. My older HP servers do much
better than this so if anyone has some similar servers and get a lot better
performance could you share your drbd.conf file so I can help figure out what
I’m missing?<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Thanks,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Morey<o:p></o:p></span></font></p>
</div>
</body>
</html>