<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="State"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="place"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Arial;
color:windowtext;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>I have experienced a failure for DRBD to reconnect after a
system forced power off/on<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>due to a hung host.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>DRBD 0.7.21<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Hearbeat 2.1.2<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>DRBD log output<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:58:06 sinfids3a1 drbd: Stopping all DRBD
resources<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:58:06 sinfids3a1 drbd: ioctl(,SET_STATE,)
failed:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>The host - on it's way down.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>At this point the filesystem was still active as Heartbeat
didn't stop successfully.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Is there a way for drbd to handle this?<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:58:06 sinfids3a1 drbd: Device or resource busy<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:58:06 sinfids3a1 drbd: Someone has opened the
device for RW access!<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:58:06 sinfids3a1 drbd: Command '/sbin/drbdsetup
/dev/drbd0 down' terminated with exit code 20<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:58:06 sinfids3a1 drbd: drbdsetup exited with
code 20<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:58:06 sinfids3a1 drbd: ERROR: Module drbd is in
use<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:58:06 sinfids3a1 drbd: .<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:58:06 sinfids3a1 rc: Stopping drbd: succeeded<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>The host - on it's way up.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:49 sinfids3a1 drbd: [<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:49 sinfids3a1 drbd: d0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:49 sinfids3a1 kernel: drbd0: Adjusting my
ra_pages to backing device's (32 -> 1024)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:49 sinfids3a1 kernel: klogd 1.4.1, ----------
state change ----------<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:49 sinfids3a1 kernel: drbd0: resync bitmap:
bits=43350764 words=677356<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:49 sinfids3a1 kernel: drbd0: size = 165 GB
(173403056 KB)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:49 sinfids3a1 kernel: mptbase: ioc0:
IOCStatus(0x0002): Busy<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:49 sinfids3a1 kernel: mptbase: ioc1:
IOCStatus(0x0002): Busy<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:49 sinfids3a1 kernel: mptbase: ioc0:
IOCStatus(0x0002): Busy<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:49 sinfids3a1 kernel: mptbase: ioc1:
IOCStatus(0x0002): Busy<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 kernel: drbd0: 180 MB marked
out-of-sync by on disk bit-map.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 kernel: drbd0: Found 6
transactions (324 active extents) in activity log.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 kernel: drbd0: Marked
additional 983 MB as out-of-sync based on <st1:State w:st="on"><st1:place
w:st="on">AL</st1:place></st1:State>.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 kernel: drbd0: drbdsetup
[5217]: cstate Unconfigured --> StandAlone<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 drbd: s0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 drbd: n0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 kernel: drbd0: drbdsetup
[5318]: cstate StandAlone --> Unconnected<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 kernel: drbd0: drbd0_receiver
[5319]: cstate Unconnected --> WFConnection<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 drbd: ]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 drbd: .<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 drbd: WARN: stdin/stdout is not
a TTY; using /dev/console<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 kernel: drbd0: using
degr_wfc_timeout=10 seconds<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: drbd0_receiver
[5319]: cstate WFConnection --> WFReportParams<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: Handshake
successful: DRBD Network Protocol version 74<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: Connection
established.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: I am(S):
1:00000002:00000001:000000c5:00000033:10<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: Peer(P):
1:00000002:00000001:000000c3:00000034:10<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: drbd0_receiver
[5319]: cstate WFReportParams --> WFBitMapS<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: sock_sendmsg
returned -32<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: drbd0_receiver
[5319]: cstate WFBitMapS --> BrokenPipe<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: short sent
ReportBitMap size=4096 sent=0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0:
Secondary/Unknown --> Secondary/Primary <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>It has detected it's partner but then breaks the
connection. This has resulted in serious data loss.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Is there a way to prevent this?<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: meta connection
shut down by peer.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: asender
terminated<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: sock was shut
down by peer<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: drbd0_receiver
[5319]: cstate BrokenPipe --> BrokenPipe<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: short read
expecting header on sock: r=0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: worker
terminated<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: drbd0_receiver
[5319]: cstate BrokenPipe --> Unconnected<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: Connection lost.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: drbd0_receiver
[5319]: cstate Unconnected --> WFConnection<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:12:01 sinfids3a1 rc: Starting drbd: succeeded<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>The scenario:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>sinfids3a1 running as master<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>host sinfids3a1 hung<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>reboot sinfids3a1 <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>sinfids3a2 running as master<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>updated filesystem (nagios config)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>powered off/on sinfids3a1<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>[.....]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>failed system over to sinfids3a1<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>updates to the filesystem were LOST !!!<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 rc: Stopping cmanic: succeeded<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 heartbeat: Stopping
High-Availability services:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 heartbeat: [5398]: info:
killing /usr/lib64/heartbeat/mgmtd -v process group 5533 with signal 15<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 mgmtd: [5533]: info: mgmtd is
shutting down<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 heartbeat: [5398]: info:
killing /usr/lib64/heartbeat/crmd process group 5532 with signal 15<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 crmd: [5532]: info: crm_shutdown:
Requesting shutdown<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 crmd: [5532]: info:
do_shutdown_req: Sending shutdown request to DC: <null><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 crmd: [5532]: info:
do_shutdown_req: Processing shutdown locally<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 crmd: [5532]: ERROR:
send_ha_message: Not connected to Heartbeat<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 crmd: [5532]: WARN:
crm_log_message_adv: #========= HA[outbound] message start ==========#<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 crmd: [5532]: WARN: MSG:
Dumping message with 10 fields<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 crmd: [5532]: WARN: MSG[0] :
[origin=do_election_vote]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 crmd: [5532]: WARN: MSG[1] :
[t=crmd]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 crmd: [5532]: WARN: MSG[2] :
[version=1.0.9]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 crmd: [5532]: WARN: MSG[3] :
[subt=request]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 crmd: [5532]: WARN: MSG[4] :
[reference=vote-crmd-1190282210-138]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 crmd: [5532]: WARN: MSG[5] :
[crm_task=vote]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 crmd: [5532]: WARN: MSG[6] :
[crm_sys_to=crmd]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 crmd: [5532]: WARN: MSG[7] :
[crm_sys_from=crmd]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 crmd: [5532]: WARN: MSG[8] :
[election-owner=3e940b7a-0e41-4df4-9279-71205d5038e8]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 crmd: [5532]: WARN: MSG[9] :
[election-id=15]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 crmd: [5532]: WARN:
send_msg_via_ha: Sending broadcast HA message (ref=vote-crmd-1190282210-138) to
crmd@<all> failed.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 crmd: [5532]: info: handle_shutdown_request:
Creating shutdown request for sinfids3a1<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:56:50 sinfids3a1 tengine: [29019]: info:
extract_event: Aborting on shutdown attribute for
3e940b7a-0e41-4df4-9279-71205d5038e8<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:57:26 sinfids3a1 sinfids3a_apache[6867]: [6959]:
INFO: 17:57:26 URL:http://0.0.0.0:80/server-status [2765/2765] ->
"-" [1]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:57:27 sinfids3a1 lrmd: [5529]: WARN:
G_SIG_dispatch: Dispatch function for SIGCHLD was delayed 1000 ms (> 100 ms)
before being called (GSource: 0x51bdc8)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:57:27 sinfids3a1 lrmd: [5529]: info:
G_SIG_dispatch: started at 455233814 should have started at 455233714<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:57:31 sinfids3a1 tengine: [29019]: info:
process_graph_event: Action resource_sinfids3a_fids_proxy_monitor_360000
initiated by a different transitioner<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:57:31 sinfids3a1 tengine: [29019]: WARN:
update_failcount: Updating failcount for resource_sinfids3a_fids_proxy on
8cc209ea-9be8-48a7-a1a1-3fdc7b7a64d4 after failed monitor: rc=7<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:57:32 sinfids3a1 tengine: [29019]: info:
extract_event: Aborting on transient_attributes changes for
8cc209ea-9be8-48a7-a1a1-3fdc7b7a64d4<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:57:32 sinfids3a1 tengine: [29019]: info:
te_update_diff: Aborting on transient_attributes deletions<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:57:33 sinfids3a1 tengine: [29019]: info:
extract_event: Aborting on transient_attributes changes for
8cc209ea-9be8-48a7-a1a1-3fdc7b7a64d4<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:57:33 sinfids3a1 tengine: [29019]: info:
te_update_diff: Aborting on transient_attributes deletions<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:57:50 sinfids3a1 rc: Stopping heartbeat: failed<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:58:06 sinfids3a1 drbd: Stopping all DRBD
resources<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:58:06 sinfids3a1 drbd: ioctl(,SET_STATE,)
failed:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:58:06 sinfids3a1 drbd: Device or resource busy<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:58:06 sinfids3a1 drbd: Someone has opened the
device for RW access!<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:58:06 sinfids3a1 drbd: Command '/sbin/drbdsetup
/dev/drbd0 down' terminated with exit code 20<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:58:06 sinfids3a1 drbd: drbdsetup exited with
code 20<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:58:06 sinfids3a1 drbd: ERROR: Module drbd is in
use<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:58:06 sinfids3a1 drbd: .<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 20 17:58:06 sinfids3a1 rc: Stopping drbd: succeeded<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:49 sinfids3a1 drbd: [<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:49 sinfids3a1 drbd: d0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:49 sinfids3a1 kernel: drbd0: Adjusting my
ra_pages to backing device's (32 -> 1024)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:49 sinfids3a1 kernel: klogd 1.4.1, ----------
state change ----------<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:49 sinfids3a1 kernel: drbd0: resync bitmap:
bits=43350764 words=677356<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:49 sinfids3a1 kernel: drbd0: size = 165 GB
(173403056 KB)<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:49 sinfids3a1 kernel: mptbase: ioc0:
IOCStatus(0x0002): Busy<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:49 sinfids3a1 kernel: mptbase: ioc1:
IOCStatus(0x0002): Busy<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:49 sinfids3a1 kernel: mptbase: ioc0:
IOCStatus(0x0002): Busy<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:49 sinfids3a1 kernel: mptbase: ioc1:
IOCStatus(0x0002): Busy<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 kernel: drbd0: 180 MB marked
out-of-sync by on disk bit-map.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 kernel: drbd0: Found 6
transactions (324 active extents) in activity log.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 kernel: drbd0: Marked
additional 983 MB as out-of-sync based on <st1:State w:st="on"><st1:place
w:st="on">AL</st1:place></st1:State>.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 kernel: drbd0: drbdsetup [5217]:
cstate Unconfigured --> StandAlone<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 drbd: s0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 drbd: n0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 kernel: drbd0: drbdsetup
[5318]: cstate StandAlone --> Unconnected<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 kernel: drbd0: drbd0_receiver
[5319]: cstate Unconnected --> WFConnection<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 drbd: ]<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 drbd: .<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 drbd: WARN: stdin/stdout is not
a TTY; using /dev/console<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:51 sinfids3a1 kernel: drbd0: using
degr_wfc_timeout=10 seconds<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: drbd0_receiver
[5319]: cstate WFConnection --> WFReportParams<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: Handshake
successful: DRBD Network Protocol version 74<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: Connection
established.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: I am(S):
1:00000002:00000001:000000c5:00000033:10<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: Peer(P):
1:00000002:00000001:000000c3:00000034:10<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: drbd0_receiver
[5319]: cstate WFReportParams --> WFBitMapS<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: sock_sendmsg
returned -32<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: drbd0_receiver
[5319]: cstate WFBitMapS --> BrokenPipe<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: short sent
ReportBitMap size=4096 sent=0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0:
Secondary/Unknown --> Secondary/Primary <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: meta connection
shut down by peer.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: asender
terminated<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: sock was shut
down by peer<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: drbd0_receiver
[5319]: cstate BrokenPipe --> BrokenPipe<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: short read
expecting header on sock: r=0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: worker
terminated<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: drbd0_receiver
[5319]: cstate BrokenPipe --> Unconnected<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: Connection lost.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a1 kernel: drbd0: drbd0_receiver
[5319]: cstate Unconnected --> WFConnection<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:12:01 sinfids3a1 rc: Starting drbd: succeeded<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:22:55 sinfids3a1 kernel: drbd0:
Secondary/Unknown --> Primary/Unknown<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:22:57 sinfids3a1 sinfids3a_Filesystem[7631]:
[7643]: INFO: Running start for /dev/drbd0 on /export<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:10:57 sinfids3a2 kernel: bcm5700: eth1 NIC Link
is Down<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:00 sinfids3a2 kernel: bcm5700: eth1 NIC Link
is Up, 1000 Mbps full duplex, receive & transmit flow control ON<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a2 kernel: drbd0: drbd0_receiver
[5318]: cstate WFConnection --> WFReportParams<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a2 kernel: drbd0: Handshake
successful: DRBD Network Protocol version 74<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a2 kernel: drbd0: Connection
established.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a2 kernel: drbd0: I am(P):
1:00000002:00000001:000000c3:00000034:10<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a2 kernel: drbd0: Peer(S):
1:00000002:00000001:000000c5:00000033:10<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a2 kernel: drbd0: Current Primary
shall become sync TARGET! Aborting to prevent data corruption.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a2 kernel: drbd0: drbd0_receiver
[5318]: cstate WFReportParams --> StandAlone<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a2 kernel: drbd0: error receiving
ReportParams, l: 72!<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a2 kernel: drbd0: asender
terminated<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a2 kernel: drbd0: worker terminated<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a2 kernel: drbd0: drbd0_receiver
[5318]: cstate StandAlone --> StandAlone<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a2 kernel: drbd0: Connection lost.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> Sep 21 10:11:53 sinfids3a2 kernel: drbd0: receiver
terminated<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
</div>
</body>
</html>