<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=iso-8859-1">
<META content="MSHTML 6.00.6000.16850" name=GENERATOR></HEAD>
<BODY text=#000000 bgColor=#ffffff>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=725295320-25112009>Actually I have those exact cards and I'm not seeing
your problem but getting those cards to work was a major pain in the rear
end. I much prefer the Myricom cards but for this HP server pair I got
stuck using the HP cards due to a political issue.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=725295320-25112009></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=725295320-25112009>Anyways, some of the things I found out about these
cards might be of help to you. We use SuSE here but doing the same for
RedHat shouldn't be much of a problem. The biggest issue is that these
cards get very hot and can over heat easily if they don't have a good amount of
airflow. Once they begin to overheat packets disapper and things fall
apart. Since you are seeing stalls after a bit of a run I would think that
you might be having an overheating issue.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=725295320-25112009></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=725295320-25112009>Also, the driver that comes with Linux kernel doesn't
work very well so you need to get the HP driver and install it. HOWEVER,
you absolutely must use the driver version that match the firmware
version. If they are different things don't work and you can't even run
the diagnostic tool. Here I'm running firmware 4.0.516 and driver
4.0.516.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=725295320-25112009></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=725295320-25112009>When I was trying to get these working I would setup
long runs of netperf and iperf and see how hot I can get the cards and then run
the diagnostic tool as it will tell you the temperature of the card. I
have found they start to freak out at about 85C. After playing around with
card position they run under load at 66C and seem to work fine with 27C ambient
air temp. </SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=725295320-25112009></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=725295320-25112009>All in all I'm not very impressed with these cards but
I got stuck using them in one place.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=725295320-25112009></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=725295320-25112009>Hope the information helps a bit,</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=725295320-25112009>Morey</SPAN></FONT></DIV><FONT face=Arial color=#0000ff
size=2></FONT><BR>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> drbd-user-bounces@lists.linbit.com
[mailto:drbd-user-bounces@lists.linbit.com] <B>On Behalf Of </B>Mike
Lovell<BR><B>Sent:</B> Wednesday, November 25, 2009 11:45 AM<BR><B>To:</B> James
Larcombe<BR><B>Cc:</B> drbd-user@lists.linbit.com<BR><B>Subject:</B> Re:
[DRBD-user] 8.3.5 Stalling on sync<BR></FONT><BR></DIV>
<DIV></DIV>hrm. i thought i had heard of someone using drbd over 10 gig with
netxen cards. i went looking for a few minutes and didn't find anything though.
my recommendation would be try newer drivers either through compiling the
drivers for you existing kernel or using a newer kernel. i don't have details on
how to do that for your cards cause i have never used any 10 gig from hp or
netxen. other than that, my only recommendation is new nics.<BR><BR>good
luck<BR><BR>mike<BR><BR>James Larcombe wrote:
<BLOCKQUOTE cite=mid:00a001ca6df3$6ffa36f0$4feea4d0$@co.uk type="cite">
<META content="Microsoft Word 12 (filtered medium)" name=Generator>
<STYLE>@font-face {
font-family: Cambria Math;
}
@font-face {
font-family: Calibri;
}
@font-face {
font-family: Tahoma;
}
@page Section1 {size: 612.0pt 792.0pt; margin: 72.0pt 72.0pt 72.0pt 72.0pt; }
P.MsoNormal {
FONT-SIZE: 11pt; MARGIN: 0cm 0cm 0pt; COLOR: black; FONT-FAMILY: "Calibri","sans-serif"
}
LI.MsoNormal {
FONT-SIZE: 11pt; MARGIN: 0cm 0cm 0pt; COLOR: black; FONT-FAMILY: "Calibri","sans-serif"
}
DIV.MsoNormal {
FONT-SIZE: 11pt; MARGIN: 0cm 0cm 0pt; COLOR: black; FONT-FAMILY: "Calibri","sans-serif"
}
A:link {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
A:visited {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlinkFollowed {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.EmailStyle17 {
COLOR: windowtext; FONT-FAMILY: "Calibri","sans-serif"; mso-style-type: personal
}
SPAN.EmailStyle18 {
COLOR: #1f497d; FONT-FAMILY: "Calibri","sans-serif"; mso-style-type: personal
}
SPAN.EmailStyle19 {
COLOR: #1f497d; FONT-FAMILY: "Calibri","sans-serif"; mso-style-type: personal-reply
}
.MsoChpDefault {
FONT-SIZE: 10pt; mso-style-type: export-only
}
DIV.Section1 {
page: Section1
}
</STYLE>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<DIV class=Section1>
<P class=MsoNormal><SPAN style="COLOR: rgb(31,73,125)">Hi
Mike,<O:P></O:P></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: rgb(31,73,125)"><O:P></O:P></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: rgb(31,73,125)">The cards I’m using are
HP NC522SFP Dual Port 10GbE Server Adapters with HP BLc 10Gb SR SFP+ Fiber
Transceivers. I could try running these with 1GB Fiber cables instead of 10GB.
<O:P></O:P></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: rgb(31,73,125)"><O:P></O:P></SPAN></P>
<P class=MsoNormal><SPAN
style="COLOR: rgb(31,73,125)">James<O:P></O:P></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: rgb(31,73,125)"><O:P></O:P></SPAN></P>
<DIV>
<DIV
style="BORDER-RIGHT: medium none; PADDING-RIGHT: 0cm; BORDER-TOP: rgb(181,196,223) 1pt solid; PADDING-LEFT: 0cm; PADDING-BOTTOM: 0cm; BORDER-LEFT: medium none; PADDING-TOP: 3pt; BORDER-BOTTOM: medium none">
<P class=MsoNormal><B><SPAN lang=EN-US
style="FONT-SIZE: 10pt; COLOR: windowtext; FONT-FAMILY: 'Tahoma','sans-serif'">From:</SPAN></B><SPAN
lang=EN-US
style="FONT-SIZE: 10pt; COLOR: windowtext; FONT-FAMILY: 'Tahoma','sans-serif'">
Mike Lovell [<A class=moz-txt-link-freetext
href="mailto:mike@dev-zero.net">mailto:mike@dev-zero.net</A>] <BR><B>Sent:</B>
25 November 2009 16:01<BR><B>To:</B> James Larcombe<BR><B>Cc:</B> <A
class=moz-txt-link-abbreviated
href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</A><BR><B>Subject:</B>
Re: [DRBD-user] 8.3.5 Stalling on sync<O:P></O:P></SPAN></P></DIV></DIV>
<P class=MsoNormal><O:P></O:P></P>
<P class=MsoNormal>nothing i tried tweaking in drbd.conf worked. the only
thing that did was changing the 10gig interfaces. what cards are you using? i
was using ones with an intel chip. the cards that i did get it to work with
were from chelsio. in my previous thread on the list, someone mentioned that
they had neterion cards working.<BR><BR>mike<BR><BR>James Larcombe wrote:
<O:P></O:P></P>
<P class=MsoNormal><SPAN style="COLOR: rgb(31,73,125)">Hi
Mike,</SPAN><O:P></O:P></P>
<P class=MsoNormal><SPAN style="COLOR: rgb(31,73,125)"></SPAN><O:P></O:P></P>
<P class=MsoNormal><SPAN style="COLOR: rgb(31,73,125)">Thanks for the quick
response. Yes you are correct we are using 10gig fibre cards. I’m not sure we
could change them though as the fibre modules used in them cost over £400
each.</SPAN><O:P></O:P></P>
<P class=MsoNormal><SPAN style="COLOR: rgb(31,73,125)"></SPAN><O:P></O:P></P>
<P class=MsoNormal><SPAN style="COLOR: rgb(31,73,125)">Is there anything I can
tweak in the drbd.conf file to get these to work.</SPAN><O:P></O:P></P>
<P class=MsoNormal><SPAN style="COLOR: rgb(31,73,125)"></SPAN><O:P></O:P></P>
<P class=MsoNormal><SPAN
style="COLOR: rgb(31,73,125)">James</SPAN><O:P></O:P></P>
<P class=MsoNormal><SPAN style="COLOR: rgb(31,73,125)"></SPAN><O:P></O:P></P>
<DIV>
<DIV
style="BORDER-RIGHT: medium none; PADDING-RIGHT: 0cm; BORDER-TOP: 1pt solid; PADDING-LEFT: 0cm; PADDING-BOTTOM: 0cm; BORDER-LEFT: medium none; PADDING-TOP: 3pt; BORDER-BOTTOM: medium none">
<P class=MsoNormal><B><SPAN lang=EN-US
style="FONT-SIZE: 10pt; COLOR: windowtext; FONT-FAMILY: 'Tahoma','sans-serif'">From:</SPAN></B><SPAN
lang=EN-US
style="FONT-SIZE: 10pt; COLOR: windowtext; FONT-FAMILY: 'Tahoma','sans-serif'">
Mike Lovell [<A href="mailto:mike@dev-zero.net"
moz-do-not-send="true">mailto:mike@dev-zero.net</A>] <BR><B>Sent:</B> 24
November 2009 17:49<BR><B>To:</B> James Larcombe<BR><B>Cc:</B> <A
href="mailto:drbd-user@lists.linbit.com"
moz-do-not-send="true">drbd-user@lists.linbit.com</A><BR><B>Subject:</B> Re:
[DRBD-user] 8.3.5 Stalling on sync</SPAN><O:P></O:P></P></DIV></DIV>
<P class=MsoNormal><O:P></O:P></P>
<P class=MsoNormal>James Larcombe wrote: <O:P></O:P></P>
<P class=MsoNormal>Hi List,<O:P></O:P></P>
<P class=MsoNormal><O:P></O:P></P>
<P class=MsoNormal>Please help. I have installed drbd 8.3.5 on Open Suse 11.1
(Kernel 2.6.27.29-0.1). <O:P></O:P></P>
<P class=MsoNormal><O:P></O:P></P>
<P class=MsoNormal>I have run drbdadm create-md dbms-test on one node and
create-md dbms-test2 on the other node. I then ran drbdadm up all on both
nodes. I then ran drbdadm -- --overwrite-data-of-my-peer primary dbms-test on
the first node and the same with dbms-test2 on the other node. They then run
for a short while before stalling. I have tried older version without success
and turning the sync rate down does not make any difference. Downing the
resources and bringing back up starts the sync again but this then stalls
quickly.<O:P></O:P></P>
<P class=MsoNormal><O:P></O:P></P>
<P class=MsoNormal>I have attached /proc/drbd, /etc/drbd.conf and a section
from /var/log/messages. Any pointers would be greatly
appreciated.<O:P></O:P></P>
<P class=MsoNormal><O:P></O:P></P>
<P class=MsoNormal>version: 8.3.5 (api:88/proto:86-91)<O:P></O:P></P>
<P class=MsoNormal>GIT-hash: ded8cdf09b0efa1460e8ce7a72327c60ff2210fb build by
root@hp-tm-40, 2009-11-24 12:21:46<O:P></O:P></P>
<P class=MsoNormal> 0: cs:SyncSource ro:Secondary/Secondary
ds:UpToDate/Inconsistent C r----<O:P></O:P></P>
<P class=MsoNormal> ns:160896 nr:0 dw:0 dr:160896 al:0 bm:9
lo:1 pe:0 ua:0 ap:0 ep:1 wo:b oos:926694296<O:P></O:P></P>
<P class=MsoNormal> [>.]
sync'ed: 0.1% (905040/905132)M
4972<O:P></O:P></P>
<P class=MsoNormal>
stalled<O:P></O:P></P>
<P class=MsoNormal> 1: cs:SyncTarget ro:Secondary/Secondary
ds:Inconsistent/UpToDate C r----<O:P></O:P></P>
<P class=MsoNormal> ns:0 nr:2173248 dw:2173248 dr:0 al:0
bm:132 lo:0 pe:29878 ua:0 ap:0 ep:1 wo:b oos:777971256<O:P></O:P></P>
<P class=MsoNormal> [>.]
sync'ed: 0.3% (759736/761856)M<O:P></O:P></P>
<P class=MsoNormal>
Stalled<O:P></O:P></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 12pt; FONT-FAMILY: 'Times New Roman','serif'"><BR>what kind
of network are you using between the two servers? this is almost the exact
same behavior i had when i was trying to get drbd to work over 10gig ethernet.
turned out to be something in drbd didn't like something about the 10gig cards
i had. i eventually had to change my network cards. what cards are you using?
1gig? 10gig? have you tried other cards? that is where i would
look.<BR><BR>mike</SPAN><O:P></O:P></P>
<P class=MsoNormal><SPAN
style="FONT-SIZE: 12pt; FONT-FAMILY: 'Times New Roman','serif'"><O:P></O:P></SPAN></P></DIV></BLOCKQUOTE><BR></BODY></HTML>