Thanks Dan - I missed that those messages were in dmesg.<div><br></div><div>So starting from 2 connected nodes (Secondary/Secondary) we set one to primary ("drbdadm primary drbd-sr1") and from my perspective my ssh connection drops and the machine locks up for about 5 minutes.</div>
<div><br></div><div>Same behavior on both nodes, either one freezes for 5 minutes when being set to the primary.. so it doesn't appear to be any kind of hardware issue specific to one of them.</div><div><br></div><div>
Below is what I'm seeing in dmesg.</div><div><br></div><div>Note - the two nodes in question are connected by a cross-over gigabit cable. </div><div><br></div><div>Very weird behavior.. after 5 minutes of freezing up the node came up again and everything seems to be ok..</div>
<div><br></div><div>Anyone have any ideas?<br><div><br></div><div><div>block drbd1: role( Secondary -> Primary ) </div><div>d-con drbd-sr1: asender terminated</div><div>d-con drbd-sr1: Terminating asender thread</div>
<div>
d-con drbd-sr1: Connection closed</div><div>block drbd1: new current UUID 5A99C51D68CDB447:188E44BA42FFFCF4:2460EA01C7EA7F96:245FEA01C7EA7F96</div><div>d-con drbd-sr1: conn( BrokenPipe -> Unconnected ) </div><div>d-con drbd-sr1: receiver terminated</div>
<div>d-con drbd-sr1: Restarting receiver thread</div><div>d-con drbd-sr1: receiver (re)started</div><div>d-con drbd-sr1: conn( Unconnected -> WFConnection ) </div><div>d-con drbd-sr1: initial packet S crossed</div><div>
d-con drbd-sr1: Handshake successful: Agreed network protocol version 101</div><div>d-con drbd-sr1: conn( WFConnection -> WFReportParams ) </div><div>d-con drbd-sr1: Starting asender thread (from drbd_r_drbd-sr1 [26469])</div>
<div>block drbd1: drbd_sync_handshake:</div><div>block drbd1: self 5A99C51D68CDB447:188E44BA42FFFCF4:2460EA01C7EA7F96:245FEA01C7EA7F96 bits:0 flags:0</div><div>block drbd1: peer 188E44BA42FFFCF4:0000000000000000:2460EA01C7EA7F96:245FEA01C7EA7F96 bits:0 flags:0</div>
<div>block drbd1: uuid_compare()=1 by rule 70</div><div>block drbd1: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> Consistent ) </div><div>block drbd1: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%</div>
<div>block drbd1: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%</div><div>block drbd1: helper command: /sbin/drbdadm before-resync-source minor-1</div><div>block drbd1: helper command: /sbin/drbdadm before-resync-source minor-1 exit code 0 (0x0)</div>
<div>block drbd1: conn( WFBitMapS -> SyncSource ) pdsk( Consistent -> Inconsistent ) </div><div>block drbd1: Began resync as SyncSource (will sync 0 KB [0 bits set]).</div><div>block drbd1: updated sync UUID 5A99C51D68CDB447:188F44BA42FFFCF4:188E44BA42FFFCF4:2460EA01C7EA7F96</div>
<div>block drbd1: Resync done (total 1 sec; paused 0 sec; 0 K/sec)</div><div>block drbd1: updated UUIDs 5A99C51D68CDB447:0000000000000000:188F44BA42FFFCF4:188E44BA42FFFCF4</div><div>block drbd1: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate ) </div>
<div><br></div><div>On Fri, Oct 5, 2012 at 6:39 PM, Dan Barker <span dir="ltr"><<a href="mailto:dbarker@visioncomm.net" target="_blank">dbarker@visioncomm.net</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">dmesg | grep sr1 should show you all you need to know.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Dan (there’s that word “should” again<g>)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:drbd-user-bounces@lists.linbit.com" target="_blank">drbd-user-bounces@lists.linbit.com</a> [mailto:<a href="mailto:drbd-user-bounces@lists.linbit.com" target="_blank">drbd-user-bounces@lists.linbit.com</a>] <b>On Behalf Of </b>Andrew Eross<br>
<b>Sent:</b> Friday, October 05, 2012 2:17 PM<br><b>To:</b> <a href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br><b>Subject:</b> [DRBD-user] IO Error Logging<u></u><u></u></span></p>
<div><div class="h5"><p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p><p class="MsoNormal" style="margin-left:.5in">Hi guys,<u></u><u></u></p><div><p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p>
</div><div><p class="MsoNormal" style="margin-left:.5in">I'm trying to debug a SSD drive that's the backing device for my secondary node.<u></u><u></u></p></div><div><p class="MsoNormal" style="margin-left:.5in">
<u></u> <u></u></p>
</div><div><p class="MsoNormal" style="margin-left:.5in">The primary/secondary are sync'd (protocol C) and everything goes fine until I get to testing fail-over, e.g.on the primary "drbdadm secondary drbd-sr1", and on the secondary "drbdadm primary drbd-sr1".<u></u><u></u></p>
</div><div><p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p></div><div><p class="MsoNormal" style="margin-left:.5in">When I do this the secondary locks up for about 5 minutes (SSH session drops) then it starts responding again and I see drbd has now dropped into diskless mode.<u></u><u></u></p>
</div><div><p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p></div><div><p class="MsoNormal" style="margin-left:.5in">I'm thinking there might be IO errors occurring with the underlying disk and perhaps drbd is automatically detaching it.<u></u><u></u></p>
</div><div><p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p></div><div><p class="MsoNormal" style="margin-left:.5in">Right now I'm running badblocks on the backing device and seeing if it can find any problems.<u></u><u></u></p>
</div><div><p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p></div><div><p class="MsoNormal" style="margin-left:.5in">In the meantime I've been trying to figure out how to get more information about IO errors from drbd.<u></u><u></u></p>
</div><div><p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p></div><div><p class="MsoNormal" style="margin-left:.5in">My devices are configured with "detach" as recommended (<a href="http://www.drbd.org/users-guide/s-configure-io-error-behavior.html" target="_blank">http://www.drbd.org/users-guide/s-configure-io-error-behavior.html</a>), however, I'm not sure how to find out more information about when this event occurs.<u></u><u></u></p>
</div><div><p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p></div><div><p class="MsoNormal" style="margin-left:.5in">Are there any debugging options I can enable that would help me see IO error details that caused a detach? <u></u><u></u></p>
</div><div><p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p></div><div><p class="MsoNormal" style="margin-left:.5in">Thanks!<u></u><u></u></p></div><div><p class="MsoNormal" style="margin-left:.5in">Andrew<u></u><u></u></p>
</div><div><p class="MsoNormal" style="margin-left:.5in"><u></u> <u></u></p></div></div></div></div></div><br>_______________________________________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" target="_blank">http://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
<br></blockquote></div><br></div></div></div>