[DRBD-user] Not able to test Automatic split brain recovery policies

Prater, James K. jprater at draper.com
Fri Apr 12 12:03:27 CEST 2013

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

I have been finding that it is better to fence and not allow the bad peer to rejoin after reboot until the original issues is determined.   Auto-rejoin has the possibility of causing a multiple flip-flopping by way of reboot cycles.  Best is to fence-reboot-analyze-rejoin, of course this requires human intervention.


----- Original Message -----
From: Digimer [mailto:lists at alteeve.ca]
Sent: Thursday, April 11, 2013 02:43 PM
To: Dan Barker <dbarker at visioncomm.net>
Cc: drbd-user at lists.linbit.com <drbd-user at lists.linbit.com>
Subject: Re: [DRBD-user] Not able to test Automatic split brain recovery	policies

On 04/11/2013 08:27 AM, Dan Barker wrote:
>> -----Original Message-----
>> From: Shailesh Vaidya [mailto:shailesh_vaidya at persistent.co.in]
>> Sent: Thursday, April 11, 2013 1:50 AM
>> To: Digimer
>> Cc: Dan Barker; drbd-user at lists.linbit.com
>> Subject: RE: [DRBD-user] Not able to test Automatic split brain recovery
>> policies
>> Hi Digimer,
>> Thanks for help and explanation. I will try it out fencing option.
>> However, I would like to validate if what I am testing for split-brain is
>> correct or not. Also what could be done for simple split-brain auto-
>> recovery through configuration without fencing.
> There is no "simple split-brain" recovery. Split Brain only occurs after an error of some sort causing two different nodes to write to the same resource while disconnected. Anything other than manual recovery of files or blocks will lose data. In many cases, it's not even possible to determine what data is being lost or how to recover it. You just have to pick the lesser of two evils and move forward, honoring the writes to one node and discarding the writes done on the other. Most applications and file systems react poorly to having writes of theirs discarded.
> Any effort spent automating the recovery of a split-brain could better be spent identifying how your configuration created the split brain, usually dual primary without sufficient controls in place to prevent split brain in the first place.
> ymmv
> Dan

To build on Dan's comments;

Automatic split-brain recovery where both nodes where StandAlone and 
Primary is not possible. Consider this;

Say you want to recover by discarding the node with the least changes;

* Node 1 has an easily replaceable ISO written to it.
* Node 2 has accounting data written to it.

A human would know to discard Node 1, obviously, but "least changes" 
would cause node 2 to get overwritten.

Say you want to recover by discarding oldest changes; Repeat the above 
example, but say that you record the accounting data an hour before the 
ISO is written. No better.

The only safe way to recover from a split-brain is to bring up the node 
you want to invalidate in StandAlone, mount the DRBD backed FS or VM, 
backup all the data to somewhere else, invalidate it, connect it to the 
still-UpToDate node and let syncing begin and then manually merge the 
just-backed up data into the now-resync'ing DRBD-backed data.

This is clumsy, prone to human errors and might well be very difficult 
or impossible, depending on the type of data stored on the DRBD resource.

*By far* the better option is to do everything you can to avoid a 
split-brain in the first place.

To test that you have accomplished that;

Setup fencing and then repeat your tests where you break the network 
connection. You should then see one node get rebooted and the remaining 
node continue. Once the fenced node powers back up, it should rejoin the 
good node without complaining about a split-brain. So if the rebooted 
node automatically rejoins, you know your configuration is working properly.

Another good test is to crash each node using 'echo c > 
/proc/sysrq-trigger'. You should see that the healthy node reboots the 
other node. If you have used a delay against a node, you should be able 
to see the difference in recovery time doing this test as well.


Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without 
access to education?
drbd-user mailing list
drbd-user at lists.linbit.com

More information about the drbd-user mailing list