[DRBD-user] possible split brain, neither node will promote to primary

Lonni J Friedman netllama at gmail.com
Thu Oct 17 21:23:33 CEST 2013

Note: "permalinks" may not be as permanent as we would like,
direct links of old sources may well be a few messages off.

Thanks for your prompt reply.  Comments/questions below....

On Thu, Oct 17, 2013 at 12:18 PM, Digimer <lists at alteeve.ca> wrote:
> On 17/10/13 15:08, Lonni J Friedman wrote:
>> [ 1727.959118] block drbd0:   state = { cs:StandAlone
>> ro:Secondary/Unknown ds:Inconsistent/DUnknown r----- }
>> [ 1728.069711] block drbd0:  wanted = { cs:StandAlone
>> ro:Primary/Unknown ds:Inconsistent/DUnknown r----- }
> This is your problem. I suspect that you invalidated the node that had
> been UpToDate. This is a pretty dangerous state to be in. If you have
> critical data on this cluster, I _strongly_ recommend calling LinBit and
> getting professional guidance. It will be worth the money.

Sadly the damage was done by a co-worker while I was out, and I'm
stuck trying to fix the mess.  The data on this cluster doesn't change
all that often, so the odds of anything being lost are quite slim.

> That said;
> The process will be a matter of deciding which node is actually UpToDate
> and then forcing it into that role. Once done, connect should work and
> the Inconsistent node should sync from UpToDate. You'll certainly want
> to do an FS consistency check after.

OK, but what's the actual command to force a node to be primary?
There used to be a -f switch to force a node to become primary, but
that options seems to no longer exist (at least in 8.4.x).

> To avoid this in the future, setup fencing (aka stonith). If you're
> using pacemaker or cman+rgmanage, you can do this easily by hooking DRBD
> into their existing fence/stonith facilities.


> In any case, please proceed very carefully. If you promote the wrong
> node, you could lose data / corrupt your FS.

More information about the drbd-user mailing list