<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:D="DAV:" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns="http://www.w3.org/TR/REC-html40"
xmlns:ns2="http://schemas.microsoft.com/sharepoint/soap/workflow/"
xmlns:ns3="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:ns1="http://schemas.microsoft.com/office/2004/12/omml"
xmlns:ns4="http://schemas.openxmlformats.org/package/2006/relationships"
xmlns:ns5="http://schemas.microsoft.com/exchange/services/2006/types"
xmlns:ns6="http://schemas.microsoft.com/exchange/services/2006/messages">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=ProgId content=Word.Document>
<meta name=Generator content="Microsoft Word 11">
<meta name=Originator content="Microsoft Word 11">
<link rel=File-List href="cid:filelist.xml@01C877C3.6D74D340">
<!--[if gte mso 9]><xml>
<o:OfficeDocumentSettings>
<o:DoNotRelyOnCSS/>
</o:OfficeDocumentSettings>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:WordDocument>
<w:SpellingState>Clean</w:SpellingState>
<w:GrammarState>Clean</w:GrammarState>
<w:DocumentKind>DocumentEmail</w:DocumentKind>
<w:EnvelopeVis/>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:BrowserLevel>MicrosoftInternetExplorer4</w:BrowserLevel>
</w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" LatentStyleCount="156">
</w:LatentStyles>
</xml><![endif]-->
<style>
<!--a:link
{mso-style-priority:99;}
span.MSOHYPERLINK
{mso-style-priority:99;}
a:visited
{mso-style-priority:99;}
span.MSOHYPERLINKFOLLOWED
{mso-style-priority:99;}
/* Font Definitions */
@font-face
{font-family:Calibri;
mso-font-alt:"Times New Roman";
mso-font-charset:0;
mso-generic-font-family:auto;
mso-font-pitch:auto;
mso-font-signature:0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{mso-style-parent:"";
margin:0in;
margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:Calibri;
mso-fareast-font-family:"Times New Roman";
mso-bidi-font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;
text-underline:single;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;
text-underline:single;}
span.EmailStyle17
{mso-style-type:personal;
mso-style-noshow:yes;
font-family:Calibri;
mso-ascii-font-family:Calibri;
mso-hansi-font-family:Calibri;
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
mso-style-noshow:yes;
mso-ansi-font-size:10.0pt;
mso-bidi-font-size:10.0pt;
font-family:Arial;
mso-ascii-font-family:Arial;
mso-hansi-font-family:Arial;
mso-bidi-font-family:Arial;
color:navy;}
span.SpellE
{mso-style-name:"";
mso-spl-e:yes;}
span.GramE
{mso-style-name:"";
mso-gram-e:yes;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;
mso-header-margin:.5in;
mso-footer-margin:.5in;
mso-paper-source:0;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-parent:"";
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-para-margin:0in;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:10.0pt;
font-family:"Times New Roman";
mso-ansi-language:#0400;
mso-fareast-language:#0400;
mso-bidi-language:#0400;}
</style>
<![endif]--><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple style='tab-interval:.5in'>
<div class=Section1>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Hi all,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>We are seeing an issue where I/O to a
volume that received an I/O error during re-sync as the sync target hangs. Looking
at the logs it seems that what’s going on is that we are skipping a <span
class=SpellE>dec_<span class=GramE>local</span></span><span class=GramE>(</span>).
<span style='mso-spacerun:yes'> </span>My theory is that <span
class=SpellE>after_state_<span class=GramE>ch</span></span><span class=GramE>(</span>)
is blocked forever waiting for <span class=SpellE>local_cnt</span> to be 0 as
we are becoming Diskless. <span style='mso-spacerun:yes'> </span>So the
worker will not do any work, hence the hang I/O. <span
style='mso-spacerun:yes'> </span>Here <span class=GramE>is the relevant logs</span>:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=1 color=black face="Courier New"><span
style='font-size:9.0pt;font-family:"Courier New";color:black'>Feb 13 03:48:55
node0 kernel: drbd5: Began <span class=SpellE>resync</span> as <span
class=SpellE>SyncTarget</span> (will sync 1048508 KB [262127 bits set]).<br>
Feb 13 03:48:55 node0 kernel: drbd5: Writing <span class=GramE>meta</span> data
super block now.<br>
Feb 13 03:48:55 node0 kernel: drbd5: Creating new epoch in <span class=SpellE>drbd_try_rs_begin_io</span><br>
Feb 13 03:48:55 node0 kernel: drbd5: ***Simulating <span class=SpellE>Resync</span>
write failure<br>
Feb 13 03:48:56 node0 kernel: drbd5: <span class=SpellE>Resync</span> aborted.<br>
Feb 13 03:48:56 node0 kernel: drbd5: <span class=SpellE><span class=GramE>conn</span></span><span
class=GramE>(</span> <span class=SpellE>SyncTarget</span> -> Connected )
disk( Inconsistent -> Failed )<br>
Feb 13 03:48:56 node0 kernel: drbd5: Local IO failed. Detaching...<br>
Feb 13 03:48:56 node0 kernel: drbd5: <span class=GramE>disk(</span> Failed
-> Diskless )<br>
Feb 13 03:48:56 node0 kernel: drbd5: Notified peer that my disk is broken.<br>
Feb 13 03:48:56 node0 kernel: drbd5: Can not write <span class=SpellE>resync</span>
data to local disk.<br>
Feb 13 03:54:57 node0 kernel: drbd5: <span class=SpellE>drbd_nl_disk_conf</span>:
<span class=SpellE>mdev</span>-><span class=SpellE><span class=GramE>bc</span></span>
not NULL.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=1 color=black face="Courier New"><span
style='font-size:9.0pt;font-family:"Courier New";color:black'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Notice the last line of the log. <span
style='mso-spacerun:yes'> </span>Our test environment must have tried to
do an “attach” so since <span class=SpellE>local_cnt</span> is not
0 we never freed the “<span class=SpellE>bc</span>”.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>But from the “</span></font><font
size=1 color=black face="Courier New"><span style='font-size:9.0pt;font-family:
"Courier New";color:black'>Can not write <span class=SpellE>resync</span> data
to local disk.” </span></font><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>we can go to <span
class=SpellE>drbd_endio_write_<span class=GramE>sec</span></span><span
class=GramE>(</span>) and there we see a suspicious :<o:p></o:p></span></font></p>
<p class=MsoNormal><span class=GramE><font size=2 color=navy face=Arial><span
style='font-size:10.0pt;font-family:Arial;color:navy'>If(</span></font></span><font
size=2 color=navy face=Arial><span style='font-size:10.0pt;font-family:Arial;
color:navy'>bio-><span class=SpellE>bi_size</span>) return 1; <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>We are supposed to do the <span
class=SpellE>dec_local</span> at the end of <span class=SpellE>drbd_endio_write</span>
<span class=GramE>sec(</span>). I am guessing that’s where the problem
is.<span style='mso-spacerun:yes'> </span>But I do not know why <span
class=SpellE>bi_size</span> would be greater then 0. <span
style='mso-spacerun:yes'> </span>Is the fix simply to <span class=SpellE>dec_local</span>
while returning?<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>BTW, the code <span class=GramE>below <span
style='mso-spacerun:yes'> </span>inserts</span> the fault in question but
won’t necessarily make the hang happens:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><span class=GramE><font size=1 color=black face="Courier New"><span
style='font-size:9.0pt;font-family:"Courier New";color:black'>#!/</span></font></span><font
size=1 color=black face="Courier New"><span style='font-size:9.0pt;font-family:
"Courier New";color:black'>bin/<span class=SpellE>sh</span><br>
<span class=GramE>echo</span> inserting fault....<br>
<span class=GramE>echo</span> 0x4 >/sys/module/<span class=SpellE>drbd/parameters/enable_faults</span><br>
<span class=GramE>echo</span> 0x20 >/sys/module/<span class=SpellE>drbd/parameters/fault_devs</span><br>
<span class=GramE>echo</span> 5 > /sys/module/<span class=SpellE>drbd/parameters/fault_rate</span><br>
<span class=GramE>echo</span> starting <span class=SpellE>resync</span><br>
<span class=GramE>sleep</span> 5<br>
<span class=SpellE><span class=GramE>drbdadm</span></span> -c /etc/<span
class=SpellE>drbd.conf.avance</span> invalidate drbd5.vol<br>
<span class=GramE>echo</span> done....<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Thanks,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>EM--<o:p></o:p></span></font></p>
</div>
</body>
</html>