<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content="text/html; charset=us-ascii" http-equiv=Content-Type>
<META name=GENERATOR content="MSHTML 8.00.6001.18999"></HEAD>
<BODY><!--ppd1000037-->
<DIV><FONT size=2 face=Verdana><SPAN class=357413002-17022011>I have several
successful DRBD clusters in production, including two RHEL 5.3 servers running
drbd 8.3.2. They have been running fine for more than a year. Today we saw very
high iowait (99%) on the primary node (possibly on the secondary too, but I
neglected to look) and users could not work. We tried finding the source of the
iowait but could not. Ended up rebooting the primary. The standby took over fine
and began a resync, but it was running very slow, like 8K per second. Resyncing
the 2TB volume was going to take 11+ hours. </SPAN></FONT></DIV>
<DIV><FONT size=2 face=Verdana><SPAN
class=357413002-17022011></SPAN></FONT> </DIV>
<DIV><FONT size=2 face=Verdana><SPAN class=357413002-17022011>So I
did...</SPAN></FONT></DIV>
<DIV><FONT size=2 face=Verdana><SPAN
class=357413002-17022011></SPAN></FONT> </DIV>
<DIV><FONT size=2 face=Verdana><SPAN class=357413002-17022011>
drbdsetup /dev/drbd0 -r 300M</SPAN></FONT></DIV>
<DIV><FONT size=2 face=Verdana><SPAN
class=357413002-17022011></SPAN></FONT> </DIV>
<DIV><FONT size=2 face=Verdana><SPAN class=357413002-17022011>The command took
more than a minute to return to the shell prompt. Now when I cat /proc/drbd, I
see the speed going at ~90K, then ~80, then ~70, then ~60, and so on, until it
reaches 0 and then it says "stalled." After 10-30 seconds in a stalled state, it
kicks back off again at about 90K, then it slowly drops back down and stalls
again.</SPAN></FONT></DIV>
<DIV><FONT size=2 face=Verdana><SPAN
class=357413002-17022011></SPAN></FONT> </DIV>
<DIV><FONT size=2 face=Verdana><SPAN class=357413002-17022011>Both servers are
on the same GigE switch. Running iperf shows that we're getting almost the full
gigabit per second on the replication link.</SPAN></FONT></DIV>
<DIV><FONT size=2 face=Verdana><SPAN
class=357413002-17022011></SPAN></FONT> </DIV>
<DIV><FONT size=2 face=Verdana><SPAN class=357413002-17022011>Any ideas how I
can troubleshoot this before users come in tomorrow morning?</SPAN></FONT></DIV>
<DIV><FONT size=2 face=Verdana></FONT> </DIV>
<P style="MARGIN: 0in 0in 0pt" class=MsoNormal align=left><SPAN
style="FONT-FAMILY: Tahoma; FONT-SIZE: 10pt"><FONT
face=verdana>--<BR></FONT></SPAN><SPAN
style="FONT-FAMILY: Tahoma; FONT-SIZE: 10pt"><FONT face=verdana>Eric
Robinson<BR></FONT></SPAN></P>
<DIV><FONT size=2 face=Tahoma></FONT> </DIV>
<DIV><FONT size=2 face=Tahoma></FONT> </DIV>
<DIV><FONT size=2 face=Tahoma></FONT> </DIV>
<DIV><FONT size=2 face=Tahoma></FONT> </DIV>
<DIV><FONT size=2 face=Tahoma></FONT> </DIV>
<DIV><FONT size=2 face=Tahoma></FONT> </DIV>
<DIV><FONT size=2 face=Tahoma></FONT> </DIV>
<DIV><FONT size=2 face=Tahoma></FONT> </DIV>
<DIV><FONT size=2 face=Tahoma></FONT> </DIV><BR><BR><FONT color=silver><SPAN style="FONT-SIZE: 7.5pt; COLOR: gray; FONT-FAMILY: Verdana"><FONT size=1>Disclaimer - February 16, 2011</FONT></FONT></SPAN></FONT> <P></P><FONT size=1><FONT face=Verdana color=gray><SPAN style="FONT-SIZE: 7.5pt; COLOR: gray; FONT-FAMILY: Verdana">This email and any files <FONT size=1>transmitted</FONT> with it are confidential and intended solely for <STRONG>drbd-user@lists.linbit.com</STRONG></SPAN></FONT><FONT face=Verdana color=gray><SPAN style="FONT-SIZE: 7.5pt; COLOR: gray; FONT-FAMILY: Verdana">. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physicians' Managed Care or Physician Select Management. Warning: Although Physicians' Managed Care or Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.</SPAN></FONT></FONT></BODY></HTML>