<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Trebuchet MS";
panose-1:2 11 6 3 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=FR link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Pascal,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>One thing is unclear : did it used to work in the past (and if yes what has changed lately that could explain this behavior) or is it a new feature you’ve just added to your customer’s config ?<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Furthermore, I suspect you have scripted all this process haven’t you ? If so, have you identified which step induces this communication disruption? Have you tried to execute manually this sequence and then at what step does it happen ?<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Best regards,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Pascal.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>De :</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> drbd-user-bounces@lists.linbit.com [mailto:drbd-user-bounces@lists.linbit.com] <b>De la part de</b> Pascal Charest<br><b>Envoyé :</b> samedi 27 août 2011 22:52<br><b>À :</b> drbd-user@lists.linbit.com<br><b>Objet :</b> [DRBD-user] Frequent disconnect when doing backup.<o:p></o:p></span></p></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-family:"Trebuchet MS","sans-serif"'>Hi, <br clear=all></span><o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>I have a small issue with one of my DRBD setup. When my backup is running (-see lower for setup and backup details), i`m getting those errors:</span><o:p></o:p></p></div><div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:24:18 pig-two -- MARK --<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:26 pig-two kernel: drbd0: peer( Secondary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )<o:p></o:p></span></p></div></div><div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:26 pig-two kernel: drbd0: asender terminated<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:26 pig-two kernel: drbd0: Terminating asender thread<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:26 pig-two kernel: drbd0: sock was reset by peer<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:26 pig-two kernel: drbd0: _drbd_send_page: size=4096 len=3064 sent=-32<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:26 pig-two kernel: drbd0: Creating new current UUID<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:26 pig-two kernel: drbd0: Writing meta data super block now.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:26 pig-two kernel: drbd0: tl_clear()<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:26 pig-two kernel: drbd0: Connection closed<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:26 pig-two kernel: drbd0: conn( NetworkFailure -> Unconnected )<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:26 pig-two kernel: drbd0: receiver terminated<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:26 pig-two kernel: drbd0: receiver (re)started<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:26 pig-two kernel: drbd0: conn( Unconnected -> WFConnection )<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:27 pig-two kernel: drbd0: Handshake successful: Agreed network protocol version 88<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:27 pig-two kernel: drbd0: Peer authenticated using 20 bytes of 'sha1' HMAC<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:27 pig-two kernel: drbd0: conn( WFConnection -> WFReportParams )<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:27 pig-two kernel: drbd0: Starting asender thread (from drbd0_receiver [3066])<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:27 pig-two kernel: drbd0: data-integrity-alg: md5<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:27 pig-two kernel: drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> UpToDate )<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:27 pig-two kernel: drbd0: Writing meta data super block now.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:27 pig-two kernel: drbd0: conn( WFBitMapS -> SyncSource ) pdsk( UpToDate -> Inconsistent )<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:27 pig-two kernel: drbd0: Began resync as SyncSource (will sync 2160 KB [540 bits set]).<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:27 pig-two kernel: drbd0: Writing meta data super block now.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:27 pig-two kernel: drbd0: Resync done (total 1 sec; paused 0 sec; 2160 K/sec)<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:27 pig-two kernel: drbd0: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:27:27 pig-two kernel: drbd0: Writing meta data super block now.<o:p></o:p></span></p></div><div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 10:44:19 pig-two -- MARK --<o:p></o:p></span></p></div></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>and<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p></div><div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:04:19 pig-two -- MARK --<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:36 pig-two kernel: drbd0: _drbd_send_page: size=4096 len=4096 sent=-104<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: peer( Secondary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: Creating new current UUID<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: Writing meta data super block now.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: asender terminated<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: Terminating asender thread<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: sock was shut down by peer<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: tl_clear()<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: Connection closed<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: conn( NetworkFailure -> Unconnected )<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: receiver terminated<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: receiver (re)started<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: conn( Unconnected -> WFConnection )<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: Handshake successful: Agreed network protocol version 88<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: Peer authenticated using 20 bytes of 'sha1' HMAC<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: conn( WFConnection -> WFReportParams )<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: Starting asender thread (from drbd0_receiver [3066])<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: data-integrity-alg: md5<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> UpToDate )<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: Writing meta data super block now.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: conn( WFBitMapS -> SyncSource ) pdsk( UpToDate -> Inconsistent )<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: Began resync as SyncSource (will sync 5788 KB [1447 bits set]).<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: Writing meta data super block now.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: Resync done (total 1 sec; paused 0 sec; 5788 K/sec)<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:20:37 pig-two kernel: drbd0: Writing meta data super block now.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Aug 27 11:44:19 pig-two -- MARK --<o:p></o:p></span></p></div></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Analysis: it look like the network is failing, then everything - under a second - re-connect, resync and work again. There are no impact on the 'production'. Anyone got some kind of idea, why ? Is it an error in my setup/design (see lower).<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p></div><div><p class=MsoNormal><b><span style='font-family:"Arial","sans-serif"'>Some background on the setup: </span></b><span style='font-family:"Arial","sans-serif"'><o:p></o:p></span></p></div><div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>It's an old version. Very old in fact - roadmap to upgrade has been drafted and submitted to client - I`m just wondering about the specific issue here... I want to be sure it's not an infrastructure design problem.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>pig-two:~# cat /proc/drbd<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>version: 8.2.6 (api:88/proto:86-88)<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>GIT-hash: 3e69822d3bb4920a8c1bfdf7d647169eba7d2eb4 build by root@pig-two, 2008-08-19 15:02:28<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'> 0: cs:Connected st:Primary/Secondary ds:UpToDate/UpToDate C r---<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'> ns:650469968 nr:0 dw:648856776 dr:16725553 al:5463958 bm:22571 lo:0 pe:0 ua:0 ap:0 oos:0<o:p></o:p></span></p></div></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>We are speaking, of:<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'> - 4x SAS 15k drives in a hardware raid-5 array (DELL Perc5)... presented to the OS as /dev/sda. <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'> - /dev/sda is the back-end device for DRBD... presented to the OS as /dev/drbd0<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'> - /dev/drbd0 is a lone "physical volume" in a volume group (called SAN) from which Logical Volume are created. Those are NOT locally mounted.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'> - those logical volumes are exported with vblade (AoE protocol, layer 2) to some other physical system (Xen dom0) where they are used as backend device (/dev/etherd/e0.1) for root volume of virtual system<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Everything work fine, but when I do backup, I follow this process: <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'> - mount a CIFS exported share over the network<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'> - take a LV snapshot, mount it, and copy everything to the CIFS share. <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'> - unmount snapshot, delete it... do for all LV.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'> - unmount network share<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>The backup are consistent and valid (tested)... What have I missed ? Should I move away from AoE to a Linux based iSCSI ? <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>P.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>--<o:p></o:p></span></p></div></div><div><p class=MsoNormal><span style='font-family:"Arial","sans-serif"'>Pascal Charest - <i>Cutting-edge technology consultant</i><br><a href="https://www.labsphoenix.com" target="_blank">https://www.labsphoenix.com</a> </span><o:p></o:p></p></div></div></body></html>