Hi, <div><br></div><div>It always `worked` - it doesn't crash. Only the communication seem to get interrupted for a few seconds while backup are being taken. Backup are valid and the setup can survive with a few seconds where redundancy is not available. </div>
<div><br></div><div>I should have asked that question when I build the setup 4 years ago, but... yeah... and now I'm trying to fix everything up for that client.</div><div><br></div><div>The broken communication seems to happen only when I'm mounting the backup snapshot and taking RAR from it. Might be a problem on the AoE side of things along with a LVM snapshot.</div>
<div><br></div><div><br></div><div>P.<br><br><div class="gmail_quote">On Sun, Aug 28, 2011 at 9:18 AM, Pascal BERTON <span dir="ltr"><<a href="mailto:pascal.berton3@free.fr">pascal.berton3@free.fr</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div lang="FR" link="blue" vlink="purple"><div><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D">Pascal,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D"><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D">One thing is unclear : did it used to work in the past (and if yes what has changed lately that could explain this behavior) or is it a new feature you’ve just added to your customer’s config ?<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D">Furthermore, I suspect you have scripted all this process haven’t you ? If so, have you identified which step induces this communication disruption? Have you tried to execute manually this sequence and then at what step does it happen ?<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D"><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D">Best regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D"><u></u> <u></u></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D">Pascal.<u></u><u></u></span></p><p class="MsoNormal">
<span lang="EN-US" style="font-size:11.0pt;color:#1F497D"><u></u> <u></u></span></p><div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"><p class="MsoNormal"><b><span style="font-size:10.0pt">De :</span></b><span style="font-size:10.0pt"> <a href="mailto:drbd-user-bounces@lists.linbit.com" target="_blank">drbd-user-bounces@lists.linbit.com</a> [mailto:<a href="mailto:drbd-user-bounces@lists.linbit.com" target="_blank">drbd-user-bounces@lists.linbit.com</a>] <b>De la part de</b> Pascal Charest<br>
<b>Envoyé :</b> samedi 27 août 2011 22:52<br><b>À :</b> <a href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br><b>Objet :</b> [DRBD-user] Frequent disconnect when doing backup.<u></u><u></u></span></p>
</div><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><span>Hi, <br clear="all"></span><u></u><u></u></p><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal"><span>I have a small issue with one of my DRBD setup. When my backup is running (-see lower for setup and backup details), i`m getting those errors:</span><u></u><u></u></p>
</div><div><div><p class="MsoNormal"><span><u></u> <u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 10:24:18 pig-two -- MARK --<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 10:27:26 pig-two kernel: drbd0: peer( Secondary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )<u></u><u></u></span></p>
</div></div><div><div><p class="MsoNormal"><span>Aug 27 10:27:26 pig-two kernel: drbd0: asender terminated<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 10:27:26 pig-two kernel: drbd0: Terminating asender thread<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 10:27:26 pig-two kernel: drbd0: sock was reset by peer<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 10:27:26 pig-two kernel: drbd0: _drbd_send_page: size=4096 len=3064 sent=-32<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 10:27:26 pig-two kernel: drbd0: Creating new current UUID<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 10:27:26 pig-two kernel: drbd0: Writing meta data super block now.<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 10:27:26 pig-two kernel: drbd0: tl_clear()<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 10:27:26 pig-two kernel: drbd0: Connection closed<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 10:27:26 pig-two kernel: drbd0: conn( NetworkFailure -> Unconnected )<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 10:27:26 pig-two kernel: drbd0: receiver terminated<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 10:27:26 pig-two kernel: drbd0: receiver (re)started<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 10:27:26 pig-two kernel: drbd0: conn( Unconnected -> WFConnection )<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 10:27:27 pig-two kernel: drbd0: Handshake successful: Agreed network protocol version 88<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 10:27:27 pig-two kernel: drbd0: Peer authenticated using 20 bytes of 'sha1' HMAC<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 10:27:27 pig-two kernel: drbd0: conn( WFConnection -> WFReportParams )<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 10:27:27 pig-two kernel: drbd0: Starting asender thread (from drbd0_receiver [3066])<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 10:27:27 pig-two kernel: drbd0: data-integrity-alg: md5<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 10:27:27 pig-two kernel: drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> UpToDate )<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 10:27:27 pig-two kernel: drbd0: Writing meta data super block now.<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 10:27:27 pig-two kernel: drbd0: conn( WFBitMapS -> SyncSource ) pdsk( UpToDate -> Inconsistent )<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 10:27:27 pig-two kernel: drbd0: Began resync as SyncSource (will sync 2160 KB [540 bits set]).<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 10:27:27 pig-two kernel: drbd0: Writing meta data super block now.<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 10:27:27 pig-two kernel: drbd0: Resync done (total 1 sec; paused 0 sec; 2160 K/sec)<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 10:27:27 pig-two kernel: drbd0: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 10:27:27 pig-two kernel: drbd0: Writing meta data super block now.<u></u><u></u></span></p></div><div><div><p class="MsoNormal"><span>Aug 27 10:44:19 pig-two -- MARK --<u></u><u></u></span></p>
</div></div><div><p class="MsoNormal"><span><u></u> <u></u></span></p></div><div><p class="MsoNormal"><span>and<u></u><u></u></span></p></div><div><p class="MsoNormal"><span><u></u> <u></u></span></p></div><div><div><p class="MsoNormal">
<span>Aug 27 11:04:19 pig-two -- MARK --<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 11:20:36 pig-two kernel: drbd0: _drbd_send_page: size=4096 len=4096 sent=-104<u></u><u></u></span></p></div><div>
<p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: peer( Secondary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )<u></u><u></u></span></p></div><div><p class="MsoNormal">
<span>Aug 27 11:20:37 pig-two kernel: drbd0: Creating new current UUID<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: Writing meta data super block now.<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: asender terminated<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: Terminating asender thread<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: sock was shut down by peer<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: tl_clear()<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: Connection closed<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: conn( NetworkFailure -> Unconnected )<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: receiver terminated<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: receiver (re)started<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: conn( Unconnected -> WFConnection )<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: Handshake successful: Agreed network protocol version 88<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: Peer authenticated using 20 bytes of 'sha1' HMAC<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: conn( WFConnection -> WFReportParams )<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: Starting asender thread (from drbd0_receiver [3066])<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: data-integrity-alg: md5<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> UpToDate )<u></u><u></u></span></p></div><div><p class="MsoNormal">
<span>Aug 27 11:20:37 pig-two kernel: drbd0: Writing meta data super block now.<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: conn( WFBitMapS -> SyncSource ) pdsk( UpToDate -> Inconsistent )<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: Began resync as SyncSource (will sync 5788 KB [1447 bits set]).<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: Writing meta data super block now.<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: Resync done (total 1 sec; paused 0 sec; 5788 K/sec)<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>Aug 27 11:20:37 pig-two kernel: drbd0: Writing meta data super block now.<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>Aug 27 11:44:19 pig-two -- MARK --<u></u><u></u></span></p>
</div></div><div><p class="MsoNormal"><span><u></u> <u></u></span></p></div><div><p class="MsoNormal"><span>Analysis: it look like the network is failing, then everything - under a second - re-connect, resync and work again. There are no impact on the 'production'. Anyone got some kind of idea, why ? Is it an error in my setup/design (see lower).<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span><u></u> <u></u></span></p></div><div><p class="MsoNormal"><span><u></u> <u></u></span></p></div><div><p class="MsoNormal"><b><span>Some background on the setup: </span></b><span><u></u><u></u></span></p>
</div><div><div><p class="MsoNormal"><span><u></u> <u></u></span></p></div><div><p class="MsoNormal"><span>It's an old version. Very old in fact - roadmap to upgrade has been drafted and submitted to client - I`m just wondering about the specific issue here... I want to be sure it's not an infrastructure design problem.<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span>pig-two:~# cat /proc/drbd<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>version: 8.2.6 (api:88/proto:86-88)<u></u><u></u></span></p></div><div><p class="MsoNormal"><span>GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by root@pig-two, 2008-08-19 15:02:28<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span> 0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r---<u></u><u></u></span></p></div><div><p class="MsoNormal"><span> ns:650469968 nr:0 dw:648856776 dr:16725553 al:5463958 bm:22571 lo:0 pe:0 ua:0 ap:0 oos:0<u></u><u></u></span></p>
</div></div><div><p class="MsoNormal"><span><u></u> <u></u></span></p></div><div><p class="MsoNormal"><span>We are speaking, of:<u></u><u></u></span></p></div><div><p class="MsoNormal"><span> - 4x SAS 15k drives in a hardware raid-5 array (DELL Perc5)... presented to the OS as /dev/sda. <u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span> - /dev/sda is the back-end device for DRBD... presented to the OS as /dev/drbd0<u></u><u></u></span></p></div><div><p class="MsoNormal"><span> - /dev/drbd0 is a lone "physical volume" in a volume group (called SAN) from which Logical Volume are created. Those are NOT locally mounted.<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span> - those logical volumes are exported with vblade (AoE protocol, layer 2) to some other physical system (Xen dom0) where they are used as backend device (/dev/etherd/e0.1) for root volume of virtual system<u></u><u></u></span></p>
</div><div><p class="MsoNormal"><span><u></u> <u></u></span></p></div><div><p class="MsoNormal"><span>Everything work fine, but when I do backup, I follow this process: <u></u><u></u></span></p></div><div><p class="MsoNormal">
<span> - mount a CIFS exported share over the network<u></u><u></u></span></p></div><div><p class="MsoNormal"><span> - take a LV snapshot, mount it, and copy everything to the CIFS share. <u></u><u></u></span></p></div>
<div><p class="MsoNormal"><span> - unmount snapshot, delete it... do for all LV.<u></u><u></u></span></p></div><div><p class="MsoNormal"><span> - unmount network share<u></u><u></u></span></p></div><div><p class="MsoNormal">
<span><u></u> <u></u></span></p></div><div><p class="MsoNormal"><span>The backup are consistent and valid (tested)... What have I missed ? Should I move away from AoE to a Linux based iSCSI ? <u></u><u></u></span></p></div>
<div><p class="MsoNormal"><span><u></u> <u></u></span></p></div><div><p class="MsoNormal"><span>P.<u></u><u></u></span></p></div><div><p class="MsoNormal"><span><u></u> <u></u></span></p></div><div><p class="MsoNormal"><span>--<u></u><u></u></span></p>
</div></div><div><p class="MsoNormal"><span>Pascal Charest - <i>Cutting-edge technology consultant</i><br><a href="https://www.labsphoenix.com" target="_blank">https://www.labsphoenix.com</a> </span><u></u><u></u></p></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>--<div>Pascal Charest -<i> Cutting-edge technology consultant</i></div><div><a href="https://labsphoenix.com" target="_blank">Les Laboratoires Phoenix</a> </div>
<br>
</div>