<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=iso-8859-7">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<style>
<!--
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Times New Roman";}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 116.0pt 1.0in 116.0pt;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>Hi Gordan and Everyone,<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>Thank you for your tips.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>Now that you mention it, this is the error I get when
the process fails<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>Apr 14 03:44:16 tweety1 kernel: drbd1: receiver
terminated<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>Apr 14 03:44:16 tweety1 kernel: drbd1: receiver
(re)started<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>Apr 14 03:44:16 tweety1 kernel: drbd1: conn(
Unconnected -> WFConnection )<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>Apr 14 03:44:16 tweety1 kernel: drbd1: Handshake
successful: Agreed network protocol version 88<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>Apr 14 03:44:16 tweety1 kernel: drbd1: Peer
authenticated using 20 bytes of 'sha1' HMAC<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>Apr 14 03:44:16 tweety1 kernel: drbd1: conn(
WFConnection -> WFReportParams )<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>Apr 14 03:44:16 tweety1 kernel: drbd1: Starting
asender thread (from drbd1_receiver [2631])<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>Apr 14 03:44:16 tweety1 kernel: drbd1:
data-integrity-alg: <not-used><o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>Apr 14 03:44:16 tweety1 kernel: drbd1: Split-Brain
detected, 2 primaries, automatically solved. Sync from peer node<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>Apr 14 03:44:16 tweety1 kernel: drbd1: <b><span
style='font-weight:bold'>helper command: /sbin/drbdadm pri-lost</span></b><o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>Apr 14 03:44:16 tweety1 kernel: drbd1: <b><span
style='font-weight:bold'>I shall become SyncTarget, but I am primary!</span></b><br>
<br>
<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>Any ideas on how to go through it?<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>Extract from my config<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>handlers {<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'> pri-on-incon-degr "echo o >
/proc/sysrq-trigger ; halt -f";<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'> pri-lost-after-sb "echo o >
/proc/sysrq-trigger ; halt -f";<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'> local-io-error "echo o >
/proc/sysrq-trigger ; halt -f";<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'> outdate-peer "/usr/lib/drbd/outdate-peer.sh
on tweety1 192.168.1.251 10.254.254.253 on tweety2 192.168.1.252
10.254.254.254";<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'> outdate-peer "/sbin/obliterate";<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'> pri-lost "echo pri-lost. Have a look at the
log files. | mail -s 'DRBD Alert' root";<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'> split-brain "echo split-brain. drbdadm --
--discard-my-data connect $DRBD_RESOURCE ? | mail -s 'DRBD Alert' root";<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>}<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>Thank you.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>-----Original Message-----<br>
From: drbd@bobich.net [mailto:drbd@bobich.net] <br>
Sent: Monday, April 14, 2008 3:23 PM<br>
To: drbd-user@lists.linbit.com<br>
Subject: Re: [DRBD-user] Question for Split Brain</span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>On Mon, 14 Apr 2008, Theophanis Kontogiannis wrote:<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>> If I have two nodes and all the resources are run
Primary/Primary, when Split Brain is<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>> dedected, and based on the algorithms, one of
them will become the sync target.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>> Let us assume that for some reason (it happens to
me once per 2-4 days), the worker fails, so<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>> the systems operate in split brain condition for
a long time.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>Sounds like you need to fix your networking problem.
Reliable <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>communication between the nodes is a pretty
fundamental requirement.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>> In that case some files have been written on A
side and some on B side.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>> Also let us assume that node A is the SyncSource
and node B is the SyncTarget.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>> <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>> This means that all files changed on node A
during the SB, will be updated on node B but the<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>> files changed in node B will not be updated on
node A?<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>Yes. Node A and node B will both end up with the
volume image from node A. <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>All changes on node B will be lost. If this is a
problem (and I can't <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>imagine it not being a problem), you should implement
fencing that will <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>forcefully shut down one of the nodes and fail over
the resources (only <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>the IP addresses if you are running two primaries) to
the remaining <o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>primary.<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'>Gordan<o:p></o:p></span></font></p>
<p class=MsoPlainText><font size=2 face="Times New Roman"><span
style='font-size:10.0pt'><o:p> </o:p></span></font></p>
</div>
</body>
</html>